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RISK 

MANAGEMENT 

GUIDE 


Acquisition  excellence  has  changed  the  way  the  Department  of  Defense  (DoD)  designs,  develops, 
manufactures,  and  supports  systems.  Our  technical,  business,  and  management  approach  for  acquiring 
and  operating  systems  has,  and  continues  to,  evolve.  For  example,  we  no  longer  can  rely  on  military 
specifications  and  standards  to  define  and  control  how  our  developers  design,  build,  and  support  our 
new  systems.  Today  we  use  commercial  hardware  and  software,  promote  open  systems  architecture, 
and  encourage  streamlining  processes,  just  to  name  a  few  of  the  initiatives  that  affect  the  way  we  do 
business.  At  the  same  time,  the  Office  of  the  Secretary  of  Defense  (OSD)  has  reduced  the  level  of 
oversight  and  review  of  programs  and  manufacturers’  plants. 

While  the  new  acquisition  model  gives  government  program  managers  and  their  contractors  broader 
control  and  more  options  than  they  have  enjoyed  in  the  past,  it  also  exposes  them  to  new  risks.  OSD 
recognizes  that  risk  is  inherent  in  any  acquisition  program  and  considers  it  essential  that  program 
managers  take  appropriate  steps  to  manage  and  control  risks. 

This  document  is  a  product  of  a  joint  effort  by  the  Under  Secretary  of  Defense  (Acquisition,  Technology 
and  Logistics  (USD  (AT&L))  staff  and  the  Defense  Acquisition  University.  It  is  based  on  the  material 
developed  by  the  DoD  Risk  Management  Working  Group.  Material  in  this  Guide  is  also  reflected  in 
the  Risk  Management  Focus  Area  of  the  Program  Management  Community  of  Practice  (PMCOP) 
(http://www.pmcop.dau.mil),  and  in  the  Defense  Acquisition  Deskbook,  which  can  be  accessed  via 
the  AT&L  Knowledge  Sharing  System  (AKSS)  Website  (http://deskbook.dau.mil/jsp/default.jsp). 


Frank  J.  Anderson,  Jr. 

President 

Defense  Acquisition  University 


PREFACE 


In  1996,  the  USD  (AT&L)  established  a  Risk  Management  Working  Group  composed  of  members  of 
the  Office  of  the  Secretary  of  Defense  (OSD)  staff,  representatives  of  the  Military  Services,  and 
members  of  other  DoD  agencies  involved  in  systems  acquisition.  This  group  reviewed  pertinent  DoD 
directives  (DoDD)  and  regulations,  examined  how  the  Services  managed  risk,  studied  various  ex¬ 
amples  of  risk  management  by  industry,  and  looked  at  DoD  training  and  education  activity  in  risk 
management.  Other  sources  of  information  were  the  Software  Engineering  Institute  Risk  Initiative, 
the  Open  Systems  Initiative,  and  the  safety  and  cost  estimating  communities.  The  findings  and  results 
of  the  Working  Group  investigation  were  presented  to  the  USD  (AT&L)  and  are  summarized  below: 

Working  Group  members  then  wrote  the  risk  management  portions  of  the  Defense  Acquisition  Desk- 
book.  The  Defense  Acquisition  Deskbook  (sometimes  referred  to  as  the  “Legacy”  Deskbook)  is  ac¬ 
cessible  from  the  AT&L  Knowledge  Sharing  System  (AKSS)  Website  (http://deskbook.dau.mil/ 
jsp/default.jsp). 


Industries 

•  Focus  of  efforts  is  to  get  a  product  to  market  at  a  competitive  price. 

•  Industry  has  have  either  a  structured  or  informal  Risk  Management  process. 

•  Evolutionary  approaches  help  avoid  or  minimize  risk. 

•  Most  approaches  employ  risk  avoidance,  early  planning,  continuous  assessment,  and  problem¬ 
solving  techniques. 

•  Structured  approaches,  when  they  exist,  are  similar  to  DoD’s  approach  to  Risk  Management. 

The  Working  Group  concluded  that  industry  has  no  magic  formula  for  Risk  Management. 

The  Military  Services 

•  The  Services  differ  in  their  approaches  to  Risk  Management. 

•  Each  approach  has  its  strengths  but  no  one  approach  is  comprehensive. 

•  Consolidation  of  the  strengths  of  each  approach  could  foster  better  Risk  Management  in  DoD. 

The  Working  Group  recommended  that  the  Defense  Acquisition  Deskbook  contain  a  set  of  guidelines 
for  sound  risk  management  practices,  and  further,  that  it  contain  a  set  of  risk  management  definitions 
that  are  comprehensive  and  useful  by  all  the  Components. 

DoD  Policy* * 

•  The  risk  management  policy  contained  in  DoDD  5000.1  is  not  comprehensive. 

The  Working  Group  recommended  that  DoDD  5000.1  be  amended  to  include  a  more  comprehensive 
set  of  risk  management  policies  that  focuses  on: 

•  The  relationship  between  the  Cost  As  an  Independent  Variable  (CAIV)  concept  and  Risk 
Management. 

•  Requirement  that  risk  management  be  prospective  (forward  looking). 

•  Establishment  of  risk  management  as  a  primary  management  technique  to  be  used  by  Program 
Managers  (PMs). 

*Note:  The  DoD  5000  policy  documents  referred  to  in  the  1996  Report  have  since  been  superseded  by  a  new  set  of  DoD 
5000  policy  and  guidance  documents  issued  in  2002-2003  time  frame. 
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DoD  Procedures 

•  Risk  Management  procedures  in  DoD  5000. 2-R  (Note:  Later  changed  to  Interim  Defense  Acquisi¬ 
tion  Guidebook)  are  inadequate  to  fully  implement  the  risk  management  policy  contained  in  DoDD 
5000.1. 

Procedures  are  lacking  regarding: 

-  Scope  of  Risk  Management 

-  Purpose  of  Risk  Management 

-  Role  of  Milestone  Decision  Authorities 

-  Risk  Management’s  support  of  CAIV 

-  Risk  assessment  during  early  acquisition  phases. 

•  Some  key  procedures  may  have  been  lost  in  transition  the  DoD  5000. 2-R,  and  need  to  be  ex¬ 
panded  upon  in  the  Defense  Acquisition  Deskbook. 

DoD  Risk  ManagementTraining 

•  Risk  management  training  for  the  DoD  Acquisition  Corps  needs  to  be  updated  and  expanded,  and 
Integrated  Product  Team  (IPT)  and  Overarching  IPT  (OIPT)  personnel  need  to  be  educated  on  the 
new  and  expanding  role  of  risk  management  in  DoD  systems  acquisition. 

•  Risk  Management  knowledge  level  needs  improvement. 

•  Education  is  a  key  to  obtaining  the  support  of  OIPTs  and  PMs.  The  Defense  Acquisition  Univer¬ 
sity  (DAU)  needs  to  include  Risk  Management  training  in  all  functional  courses  and  develop  a 
dedicated  risk  management  course  for  acquisition  corps  personnel. 


The  recommendations  of  the  Risk  Management  Working  Group  have  been  fully  implemented  over  the 
period  1 996-2003 .  The  Risk  Management  part  of  the  Defense  Acquisition  Deskbook  and  material  in  the 
Risk  Focus  Area  of  the  Program  Management  Community  of  Practice  (PMCoP)  (http:// 
www.pmcop.dau.mil)  form  the  basis  for  this  Guide.  The  goal  of  the  Risk  Management  Guide  is  to 
provide  acquisition  professionals  and  program  management  offices  with  a  practical  reference  for  deal¬ 
ing  with  system  acquisition  risks.  It  has  also  been  designed  to  be  used  as  an  aid  in  DAU  course  offerings. 

This  Guide  reflects  the  efforts  of  many  people.  Mr.  Mark  Schaeffer,  former  Deputy  Director,  Systems 
Engineering,  who  chaired  the  initial  Risk  Management  Working  Group,  and  Mr.  Mike  Zsak  and  Mr. 
Tom  Parry,  formerly  from  the  AT&L  Systems  Engineering  Support  Office,  were  the  original  driving 
forces  behind  the  risk  management  initiative.  LtCol  John  Driessnack,  US  AF,  from  the  DAU/DSMC 
faculty;  Mr.  Greg  Caruth,  Ms.  Debbie  Gonzalez,  and  Ms.  Frances  Battle  from  the  DAU  Press;  and 
Ms.  Patricia  Bartlett  from  Bartlett  Communications  guided  the  composition  of  the  Guide.  Assistance 
was  also  provided  by  Mr.  Jeff  Turner  of  the  DAU  Publications  Distribution  Center.  Special  recogni¬ 
tion  goes  to  the  Institute  for  Defense  Analyses  team  composed  of  Mr.  Louis  Simpleman,  Mr.  Ken 
Evans,  Mr.  Jim  Lloyd,  Mr.  Gerald  Pike,  and  Mr.  Richard  Roemer,  who  compiled  the  data  and  wrote 
major  portions  of  the  text.  Also  special  thanks  to  Ms.  Margaret  Adcock  for  her  detailed  comments  and 
support,  and  to  Dr.  Edmund  Conrow  for  his  suggestions  and  recommendations  that  have  vastly  improved 
the  Guide. 


Charles  B.  Cochrane 
Director 

DAU  Center  for  Program  Management 
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William  W.  Bahnmaier 
Editor 
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INTRODUCTION 


Risk  has  always  been  a  concern  in  the  acquisi¬ 
tion  of  Department  of  Defense  (DoD)  systems. 
The  acquisition  process  itself  is  designed,  to  a 
large  degree,  to  allow  risks  to  be  controlled  from 
conception  to  delivery  of  a  system.  Unfortu¬ 
nately,  in  the  past,  some  Program  Managers 
(PMs)  and  decision  makers  have  viewed  risk 
as  something  to  be  avoided.  Any  program  that 
had  risk  was  subject  to  intense  review  and  over¬ 
sight.  This  attitude  has  changed.  DoD  manag¬ 
ers  recognize  that  risk  is  inherent  in  any  pro¬ 
gram  and  that  it  is  necessary  to  analyze  future 
program  events  to  identify  potential  risks  and 
take  measures  to  handle  them. 

Risk  management  is  concerned  with  the  out¬ 
come  of  future  events,  whose  exact  outcome  is 
unknown,  and  with  how  to  deal  with  these  un¬ 
certainties,  i.e.,  a  range  of  possible  outcomes. 
In  general,  outcomes  are  categorized  as  favor¬ 
able  or  unfavorable,  and  risk  management  is 
the  art  and  science  of  planning,  assessing,  and 
handling  future  events  to  ensure  favorable  out¬ 
comes.  The  alternative  to  risk  management  is 
crisis  management,  a  resource-intensive  process 
that  is  normally  constrained  by  a  restricted  set 
of  available  options. 

1.1  PURPOSE  AND  SCOPE 

This  Risk  Management  Guide  is  designed  to 
provide  acquisition  professionals  and  program 
management  offices  (PMOs)  with  apractial  ref¬ 
erence  book  for  dealing  with  system  acquisition 
risks.  It  is  also  intended  to  be  useful  as  an  aid  in 


classroom  instruction  and  as  a  reference  book  for 
practical  applications.  Most  of  the  material  in  this 
Guide  is  derived  from  the  Defense  Acquisition 
Deskbook  of  the  Acquisition,  Technology,  and 
Logistics  (AT&L)  Knowledge  Sharing  System 
(AKSS)  and  from  the  Risk  Focus  Area  of  the 
Program  Management  Community  of  Practice 
(PMCoP).  Readers  should  refer  to  the  PMCoP 
Website  (http://www.pmcop.dau.  mil)  for  any 
new  risk  management  information  that  is  dissemi¬ 
nated  between  publishing  of  updated  Guide  edi¬ 
tions  or  versions  of  editions. 

1.2  ORGANIZATION  OF  THE  GUIDE 

The  Risk  Management  Guide  discusses  risk  and 
risk  management,  defines  terms,  and  introduces 
basic  risk  management  concepts  (Chapter  2). 

Chapter  3  examines  risk  management  concepts 
relative  to  the  DoD  acquisition  process.  It 
illustrates  how  risk  management  is  an  integral 
part  of  program  management,  describes  interac¬ 
tion  with  other  acquisition  processes,  and  iden¬ 
tifies  and  discusses  the  various  types  of  acquisi¬ 
tion  risks. 

Chapter  4  discusses  the  implementation  of  a  risk 
management  program  from  the  perspective  of  a 
PMO.  This  chapter  focuses  on  practical  appli¬ 
cation  issues  such  as  risk  management  program 
design  options,  PMO  risk  management  organi¬ 
zations,  and  criteria  for  a  risk  management  in¬ 
formation  system  (MIS). 


1 


Chapter  5,  the  final  chapter,  describes  a  number 
of  techniques  that  address  the  aspects  (phases) 
of  risk  management,  i.e.,  planning,  assessment, 
handling,  and  monitoring. 

This  Guide  is  a  source  of  background  informa¬ 
tion  and  provides  a  starting  point  for  a  risk  man¬ 
agement  program.  None  of  the  material  is  man¬ 
datory.  PMs  should  tailor  the  approaches  and 
techniques  to  fit  their  programs. 

The  Risk  Management  Guide  also  contains 
appendices  that  are  intended  to  serve  as  refer¬ 
ence  material  and  examples,  and  provide 
backup  detail  for  some  of  the  concepts  pre¬ 
sented  in  the  main  portion  of  the  Guide. 

1.3  APPROACH  TO  RISK 
MANAGEMENT 

Based  on  the  DoD  model  contained  in  the  De¬ 
fense  Acquisition  Deskbook  (described  in  Chap¬ 
ter  2),  this  Guide  emphasizes  a  risk  management 
approach  that  is  disciplined,  forward  looking,  and 
continuous. 

In  1986,  the  Government  Accounting  Office 
(GAO),  as  part  of  an  evaluation  of  DoD  poli¬ 
cies  and  procedures  for  technical  risk  assess¬ 
ments,  developed  a  set  of  criteria  as  an  approach 
to  good  risk  assessments.  These  criteria,  with 
slight  modification,  apply  to  all  aspects  of  risk 
management  and  are  encompassed  in  the 
Guide’s  approach.  They  are: 

(1)  Planned  Procedures.  Risk  management 
is  planned  and  systematic. 

(2)  Prospective  Assessment.  Potential  future 
problems  are  considered,  not  just  current 
problems. 

(3)  Attention  to  Technical  Risk.  There  is 
explicit  attention  to  technical  risk. 


(4)  Documentation.  All  aspects  of  the  risk  man¬ 
agement  program  are  recorded  and  data 
maintained. 

(5)  Continual  Process.  Risk  assessments  are 
made  throughout  the  acquisition  process; 
handling  activities  are  continually  evaluated 
and  changed  if  necessary;  and  critical  risk 
areas  are  always  monitored. 

While  these  criteria  are  not  solely  sufficient  to 
determine  the  “health”  of  a  program,  they  are 
important  indicators  of  how  well  a  risk 
management  process  is  being  implemented.  A 
pro-active  risk  management  process  is  a  good 
start  toward  a  successful  program. 

1.4  DOD  RISK  MANAGEMENT 
POLICIES  AND  PROCEDURES 

DoD  policies  and  procedures  that  address  risk 
management  for  acquisition  programs  are  con¬ 
tained  in  five  key  DoD  documents.  DoD  Direc¬ 
tive  (DoDD)  5000. 1  ( The  Defense  Acquisition 
System)  contains  overall  acquisition  policy  — 
with  a  strong  basis  in  risk  management.  The 
policy  on  risk  management  is  amplified  further 
by  the  information  in  DoD  Instruction  (DoDI) 
5000.2  ( Operation  of  the  Defense  Acquisition 
System )  and  the  Interim  Defense  Acquisition 
Guidebook  (IDAG).  These  documents  integrate 
risk  management  into  the  acquisition  process,  de¬ 
scribe  the  relationship  between  risk  and  various 
acquisition  functions,  and  establish  some 
reporting  requirements.  DoDD  5000.4  and  DoD 
5000.4-M  address  risk  and  cost  analysis  guidance 
as  they  apply  to  the  Office  of  the  Secretary  of 
Defense.  Appendix  A  is  an  extract  of  existing 
risk  management  policies  and  procedures  from 
all  of  these  documents. 

The  DoD  5000  series  contains  strong  statements 
on  risk  management  but  requires  elaboration  to 
help  the  PM  establish  an  effective  risk  manage¬ 
ment  program.  The  information  furnished  in  the 
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Risk  Management  section  of  the  Defense  Ac¬ 
quisition  Deskbook  and  in  the  Risk  Focus  Area 
of  the  PMCoP  supports  and  expands  the  con¬ 
tents  of  the  DoD  5000  series.  This  Guide  in  turn 
is  derived  from  and  reflects  those  sources. 
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2 

RISK  AND 

RISK  MANAGEMENT 


2.1  INTRODUCTION 

This  Chapter  introduces  the  concepts  of  risk  and 
risk  management  by  explaining  the  DoD  risk- 
related  definitions  and  by  identifying  the  char¬ 
acteristics  of  acquisition  risks.  It  also  presents 
and  discusses  a  structured  concept  for  risk 
management  and  its  five  subordinate  processes. 

2.2  OVERVIEW 

The  DoD  risk  management  concept  is  based  on 
the  principles  that  risk  management  must  be 
forward-looking,  structured,  informative,  and 
continuous.  The  key  to  successful  risk  manage¬ 
ment  is  early  planning  and  aggressive  execu¬ 
tion.  Good  planning  enables  an  organized,  com¬ 
prehensive,  and  iterative  approach  for  identi¬ 
fying  and  assessing  the  risk  and  handling  op¬ 
tions  necessary  to  refine  a  program  acquisition 
strategy.  To  support  these  efforts,  assessments 
should  be  performed  as  early  as  possible  in  the 
life  cycle  to  ensure  that  critical  technical,  sched¬ 
ule,  and  cost  risks  are  addressed  with  handling 
actions  incorporated  into  program  planning  and 
budget  projections. 

PMs  should  update  program  risk  assessments 
and  tailor  their  management  strategies  accord¬ 
ingly.  Early  information  gives  them  data  that 
helps  when  writing  a  Request  for  Proposal  and 
assists  in  Source  Selection  planning.  As  a  pro¬ 
gram  progresses,  new  information  improves 


insight  into  risk  areas,  thereby  allowing  the  de¬ 
velopment  of  effective  handling  strategies.  The 
net  result  promotes  executable  programs. 

Effective  risk  management  requires  involve¬ 
ment  of  the  entire  program  team  and  also  re¬ 
quires  help  from  outside  experts  knowledge¬ 
able  in  critical  risk  areas  (e.g.,  threat,  technol¬ 
ogy,  design,  manufacturing,  logistics,  schedule, 
and  cost).  In  addition,  the  risk  management  pro¬ 
cess  should  cover  hardware,  software,  the  hu¬ 
man  element,  and  integration  issues.  Outside 
experts  may  include  representatives  from  the 
user,  laboratories,  contract  management,  test, 
logistics,  and  sustainment  communities,  and 
industry.  Users,  essential  participants  in  pro¬ 
gram  trade  analyses,  should  be  part  of  the  as¬ 
sessment  process  so  that  an  acceptable  balance 
among  cost,  schedule,  performance,  and  risk 
can  be  reached.  A  close  relationship  between 
the  Government  and  industry,  and  later  with  the 
selected  contractor(s),  promotes  an  understand¬ 
ing  of  program  risks  and  assists  in  developing 
and  executing  the  management  efforts. 

Successful  risk  management  programs  gen¬ 
erally  have  the  following  characteristics: 

€  Feasible,  stable,  and  well-understood  user 
requirements  and  threat; 

•  A  close  relationship  with  user,  industry,  and 
other  appropriate  participants; 
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•  A  planned  and  structured  risk  management 
process,  integral  to  the  acquisition  process; 

•  An  acquisition  strategy  consistent  with  risk 
level  and  risk-handling  strategies; 

•  Continual  reassessment  of  program  and 
associated  risks; 

•  A  defined  set  of  success  criteria  for  all  cost, 
schedule,  and  performance  elements,  e.g., 
Acquisition  Program  Baseline  (APB) 
thresholds; 

•  Metrics  to  monitor  effectiveness  of  risk¬ 
handling  strategies; 

•  Effective  Test  and  Evaluation  Program;  and 

•  Formal  documentation. 

PMs  should  follow  the  guidelines  below  to 

ensure  that  a  management  program  possesses 

the  above  characteristics. 

•  Assess  program  risks,  using  a  structured  pro¬ 
cess,  and  develop  strategies  to  manage  these 
risks  throughout  each  acquisition  phase. 

•  Identify  early  and  intensively  manage  those 
design  parameters  that  critically  affect  cost, 
capability,  or  readiness. 

•  Use  technology  demonstrations/modeling/ 
simulation  and  aggressive  prototyping  to 
reduce  risks. 

•  Use  test  and  evaluation  as  a  means  of 
quantifying  the  results  of  the  risk-handling 
process. 

•  Include  industry  and  user  participation  in  risk 
management. 


•  Use  Developmental  Test  and  Evaluation 
(DT&E)  and  early  operational  assessments 
when  appropriate. 

•  Establish  a  series  of  “risk  assessment  re¬ 
views”  to  evaluate  the  effectiveness  of  risk 
handling  against  clearly  defined  success 
criteria. 

•  Establish  the  means  and  format  to  communi¬ 
cate  risk  information  and  to  train  participants 
in  risk  management. 

•  Prepare  an  assessment  training  package  for 
members  of  the  program  office  and  others, 
as  needed. 

•  Acquire  approval  of  accepted  risks  at  the 
appropriate  decision  level. 

In  general,  management  of  software  risk  is  the 
same  as  management  of  other  types  of  risk 
and  techniques  that  apply  to  hardware  programs 
are  equally  applicable  to  software  intensive 
programs.  Nevertheless,  some  characteristics  of 
software  make  this  type  of  risk  management  dif¬ 
ferent,  primarily  because  it  is  difficult  to: 

•  Identify  software  risk. 

•  Estimate  the  time  and  resources  required  to 
develop  new  software,  resulting  in  potential 
risks  in  cost  and  schedule. 

•  Test  software  completely  because  of  the 
number  of  paths  that  can  be  followed  in  the 
logic  of  the  software. 

•  Develop  new  programs  because  of  the  rapid 
changes  in  information  technology  and  an 
ever-increasing  demand  for  quality  software 
personnel. 
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2.3  RISK  MANAGEMENT 

STRUCTURE  AND  DEFINITIONS 

Although  each  risk  management  strategy 
depends  upon  the  nature  of  the  system  being 
developed,  research  reveals  that  good  strategies 
contain  the  same  basic  processes  and  structure 
shown  in  Figure  2-1.  This  structure  is  some¬ 
times  also  referred  to  as  the  Risk  Management 
Process  Model.  The  application  of  these  pro¬ 
cesses  vary  with  acquisition  phases  and  the  de¬ 
gree  of  system  definition;  all  should  be  inte¬ 
grated  into  the  program  management  function. 
The  elements  of  the  structure  are  discussed  in 
the  following  paragraphs  of  this  Chapter;  how¬ 
ever,  in  order  to  form  a  basis  for  discussion, 
the  Defense  Acquisition  Deskbook  definitions  for 
the  processes  and  elements  of  risk  management 
include: 

Risk  is  a  measure  of  the  potential  inability  to 
achieve  overall  program  objectives  within  de¬ 
fined  cost,  schedule,  and  technical  constraints 
and  has  two  components:  (1)  the  probability/ 
likelihood  of  failing  to  achieve  a  particular  out¬ 
come,  and  (2)  the  consequences/impacts  of  fail¬ 
ing  to  achieve  that  outcome. 


Risk  events,  i.e.,  things  that  could  go  wrong  for 
a  program  or  system,  are  elements  of  an  acquisi¬ 
tion  program  that  should  be  assessed  to  deter¬ 
mine  the  level  of  risk.  The  events  should  be  de¬ 
fined  to  a  level  that  an  individual  can  compre¬ 
hend  the  potential  impact  and  its  causes.  For  ex¬ 
ample,  a  potential  risk  event  for  a  turbine  engine 
could  be  turbine  blade  vibration.  There  could 
be  a  series  of  potential  risk  events  that  should  be 
selected,  examined,  and  assessed  by  subject- 
matter  experts. 

The  relationship  between  the  two  components 
of  risk  —  probability  and  consequence/impact 
—  is  complex.  To  avoid  obscuring  the  results  of 
an  assessment,  the  risk  associated  with  an  event 
should  be  characterized  in  terms  of  its  two  com¬ 
ponents.  As  part  of  the  assessment  there  is  also 
a  need  for  backup  documentation  containing 
the  supporting  data  and  assessment  rationale. 

Risk  management  is  the  act  or  practice  of  deal¬ 
ing  with  risk.  It  includes  planning  for  risk,  as¬ 
sessing  (identifying  and  analyzing)  risk  areas, 
developing  risk-handling  options,  monitoring 
risks  to  determine  how  risks  have  changed,  and 
documenting  the  overall  risk  management 
program. 
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Risk  planning  is  the  process  of  developing  and 
documenting  an  organized,  comprehensive,  and 
interactive  strategy  and  methods  for  identify¬ 
ing  and  tracking  risk  areas,  developing  risk¬ 
handling  plans,  performing  continuous  risk  as¬ 
sessments  to  determine  how  risks  have  changed, 
and  assigning  adequate  resources. 

Risk  assessment  is  the  process  of  identifying 
and  analyzing  program  areas  and  critical  tech¬ 
nical  process  risks  to  increase  the  probability/ 
likelihood  of  meeting  cost,  schedule,  and  per¬ 
formance  objectives.  Risk  identification  is  the 
process  of  examining  the  program  areas  and 
each  critical  technical  process  to  identify  and 
document  the  associated  risk.  Risk  analysis  is 
the  process  of  examining  each  identified  risk 
area  or  process  to  refine  the  description  of  the 
risk,  isolating  the  cause,  and  determining  the 
effects.  It  includes  risk  rating  and  prioritization 
in  which  risk  events  are  defined  in  terms  of  their 
probability  of  occurrence,  severity  of  conse¬ 
quence/impact,  and  relationship  to  other  risk 
areas  or  processes. 

Risk  handling  is  the  process  that  identifies, 
evaluates,  selects,  and  implements  options  in 
order  to  set  risk  at  acceptable  levels  given  pro¬ 
gram  constraints  and  objectives.  This  includes 
the  specifics  on  what  should  be  done,  when  it 
should  be  accomplished,  who  is  responsible, 
and  associated  cost  and  schedule.  The  most  ap¬ 
propriate  strategy  is  selected  from  these  han¬ 
dling  options.  For  purposes  of  the  Guide ,  risk 
handling  is  an  all-encompassing  term  whereas 
risk  mitigation  is  one  subset  of  risk  handling. 

Risk  monitoring  is  the  process  that  systemati¬ 
cally  tracks  and  evaluates  the  performance  of 
risk-handling  actions  against  established 
metrics  throughout  the  acquisition  process  and 
develops  further  risk-handling  options,  as 
appropriate.  It  feeds  information  back  into  the 
other  risk  management  activities  of  planning, 
assessment,  and  handling  as  shown  in  Figure 


2-1.  This  feedback  mechanism  was  first  sug¬ 
gested  by  Dr.  Edmund  Conrow  in  his  book 
Effective  Risk  Management:  Some  Keys  to 
Success. 

Risk  documentation  is  recording,  maintaining, 
and  reporting  assessments,  handling  analysis 
and  plans,  and  monitoring  results.  It  includes 
all  plans,  reports  for  the  PM  and  decision 
authorities,  and  reporting  forms  that  may  be 
internal  to  the  PMO. 

2.4  RISK  DISCUSSION 

Implicit  in  the  definition  of  risk  is  the  concept 
that  risks  are  future  events  ,  i.e.,  potential  prob¬ 
lems,  and  that  there  is  uncertainty  associated 
with  the  program  if  these  risk  events  occur. 
Therefore,  there  is  a  need  to  determine,  as  much 
as  possible,  the  probability  of  a  risk  event 
occurring  and  to  estimate  the  consequence/ 
impact  if  it  occurs.  The  combination  of  these  two 
factors  determines  the  level  of  risk.  For  example, 
an  event  with  a  low  probability  of  occurring,  yet 
with  severe  consequences/impacts,  may  be  a  can¬ 
didate  for  handling.  Conversely,  an  event  with  a 
high  probability  of  happening,  but  the  conse¬ 
quences/impacts  of  which  do  not  affect  a 
program,  may  be  acceptable  and  require  no 
handling. 

To  reduce  uncertainty  and  apply  the  definition 
of  risk  to  acquisition  programs,  PMs  must  be 
familiar  with  the  types  of  acquisition  risks,  un¬ 
derstand  risk  terminology,  and  know  how  to 
measure  risk.  These  topics  are  addressed  in  the 
next  several  sections. 

2.4.1  Characteristics  of  Acquisition  Risk 

Acquisition  programs  tend  to  have  numerous, 
often  interrelated,  risks.  They  are  not  always 
obvious;  relationships  may  be  obscure;  and  they 
may  exist  at  all  program  levels  throughout  the 
life  of  a  program.  Risks  are  in  the  PMO  (program 
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plans,  etc.);  in  support  provided  by  other  Gov¬ 
ernment  agencies;  in  threat  assessment;  and  in 
prime  contractor  processes,  engineering  and 
manufacturing  processes,  and  technology.  The 
interrelationship  among  risk  events  may  cause 
an  increase  in  one  because  of  the  occurrence  of 
another.  For  example,  a  slip  in  schedule  for  an 
early  test  event  may  adversely  impact  subse¬ 
quent  tests,  assuming  a  fixed  period  of  test  time 
is  available. 

Another  important  risk  characteristic  is  the  time 
period  before  a  risk  future  event  occurs;  because 
time  is  critical  in  determining  risk-handling 
options.  If  an  event  is  imminent,  the  PMO  must 
resort  to  crisis  management.  An  event  that  is 
far  enough  in  the  future  to  allow  management 
actions  may  be  controllable.  The  goal  is  to  avoid 
the  need  to  revert  to  crisis  management  and 
problem  solving  by  managing  risk  up  front. 

An  event’s  probability  of  occurrence  and  con¬ 
sequences/impacts  may  change  as  the  develop¬ 
ment  process  proceeds  and  information  be¬ 
comes  available.  Therefore,  throughout  the  de¬ 
velopment  phase,  PMOs  should  reevaluate 
known  risks  on  a  periodic  basis  and  examine 
the  program  for  new  risks. 

2.4.2  Program  Products,  Processes, 

Risk  Areas,  and  Risk  Events 

Program  risk  includes  all  risk  events  and  their 
relationships  to  each  other.  It  is  a  top-level  as¬ 
sessment  of  impact  to  the  program  when  all  risk 
events  at  the  lower  levels  of  the  program  are 
considered.  Program  risk  may  be  a  roll-up  of 
all  low-level  events;  however,  most  likely,  it  is 
a  subjective  evaluation  of  the  known  risks  by 
the  PMO,  based  on  the  judgment  and  experi¬ 
ence  of  experts.  Any  roll-up  of  program  risks 
must  be  carefully  done  to  prevent  key  risk  issues 
from  “slipping  through  the  cracks.”  Identify¬ 
ing  program  risk  is  essential  because  it  forces 
the  PMO  to  consider  relationships  among  all 


risks  and  may  identify  potential  areas  of  concern 
that  would  have  otherwise  been  overlooked. 
One  of  the  greatest  strengths  of  a  formal,  con¬ 
tinuous  risk  management  process  is  the  proac¬ 
tive  quest  to  identify  risk  events  for  handling  and 
the  reduction  of  uncertainty  that  results  from  han¬ 
dling  actions. 

A  program  office  has  continuous  demands  on 
its  time  and  resources.  It  is,  at  best,  difficult, 
and  probably  impossible,  to  assess  every 
potential  area  and  process.  To  manage  risk,  the 
PMOs  should  focus  on  the  critical  areas  that 
could  affect  the  outcome  of  their  programs. 
Work  Breakdown  Structure  (WBS)  product  and 
process  elements  and  industrial  engineering 
and  manufacturing  processes  contain  most  of 
the  significant  risk  events.  Risk  events  are  de¬ 
termined  by  examining  each  WBS  element  and 
process  in  terms  of  sources  or  areas  of  risk. 
Broadly  speaking,  these  sources  generally  can 
be  grouped  as  cost,  schedule,  and  performance, 
with  the  latter  including  technical  risk. 
Following  are  some  typical  risk  areas: 

•  Threat.  The  sensitivity  of  the  program  to 
uncertainty  in  the  threat  description,  the 
degree  to  which  the  system  design  would 
have  to  change  if  the  threat’s  parameters 
change,  or  the  vulnerability  of  the  program 
to  foreign  intelligence  collection  efforts 
(sensitivity  to  threat  countermeasure). 

•  Requirements.  The  sensitivity  of  the  program 
to  uncertainty  in  the  system  description  and 
requirements  except  for  those  caused  by 
threat  uncertainty. 

•  Design.  The  ability  of  the  system  configu¬ 
ration  to  achieve  the  program’s  engineering 
objectives  based  on  the  available  technology, 
design  tools,  design  maturity,  etc. 

•  Test  and  Evaluation  (T&E) .  The  adequacy 
and  capability  of  the  T&E  program  to  assess 
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attainment  of  significant  performance  speci¬ 
fications  and  determine  whether  the  systems 
are  operationally  effective  and  suitable. 

•  Modeling  and  Simulation  (M&S) .  The  ad¬ 
equacy  and  capability  of  M&S  to  support  all 
phases  of  a  program  using  verified,  valid, 
and  accredited  M&S  tools. 

•  Technology.  The  degree  to  which  the  tech¬ 
nology  proposed  for  the  program  has  been 
demonstrated  as  capable  of  meeting  all  of 
the  program’s  objectives. 

•  Logistics.  The  ability  of  the  system  configu¬ 
ration  to  achieve  the  program’s  logistics  ob¬ 
jectives  based  on  the  system  design,  main¬ 
tenance  concept,  support  system  design,  and 
availability  of  support  resources. 

•  Production.  The  ability  of  the  system  con¬ 
figuration  to  achieve  the  program’s  produc¬ 
tion  objectives  based  on  the  system  design, 
manufacturing  processes  chosen,  and  avail¬ 
ability  of  manufacturing  resources  such  as 
facilities  and  personnel. 

•  Concurrency.  The  sensitivity  of  the  pro¬ 
gram  to  uncertainty  resulting  from  the  com¬ 
bining  or  overlapping  of  life-cycle  phases  or 
activities. 

•  Capability  of  Developer.  The  ability  of  the 
developer  to  design,  develop,  and  manufac¬ 
ture  the  system.  The  contractor  should  have 
the  experience,  resources,  and  knowledge  to 
produce  the  system. 

•  Cost/Funding.  The  ability  of  the  system  to 
achieve  the  program’s  life-cycle  cost  objec¬ 
tives.  This  includes  the  effects  of  budget  and 
affordability  decisions  and  the  effects  of 
inherent  errors  in  the  cost  estimating 
technique(s)  used  (given  that  the  technical 
requirements  were  properly  defined). 


•  Management.  The  degree  in  which  program 
plans  and  strategies  exist  and  are  realistic  and 
consistent.  The  Government’s  acquisition 
team  should  be  qualified  and  sufficiently 
staffed  to  manage  the  program. 

•  Schedule.  The  adequacy  of  the  time  allocated 
for  performing  the  defined  tasks,  e.g.,  develop¬ 
mental,  production,  etc.  This  factor  includes  the 
effects  of  programmatic  schedule  decisions,  the 
inherent  errors  in  the  schedule  estimating  tech¬ 
nique  used,  and  external  physical  constraints. 

Critical  risk  processes  are  the  developer’s  engi¬ 
neering  and  production  processes  which,  histori¬ 
cally,  have  caused  the  most  difficulty  during  the 
development  and/or  production  phases  of  acqui¬ 
sition  programs.  These  processes  include,  but  are 
not  limited  to,  design,  test,  production,  facili¬ 
ties,  logistics,  and  management.  These  pro¬ 
cesses  are  included  in  the  critical  risk  areas  and 
are  addressed  separately  to  emphasize  that  they 
focus  on  processes.  DoD  4245. 7-M,  Transition 
from  Development  to  Production,  describes  them 
using  templates.  See  Figure  2-2  for  an  example 
of  the  template  for  product  development.  The 
templates  are  the  result  of  a  Defense  Science 
Board  task  force,  composed  of  Government  and 
industry  experts,  who  identified  engineering  pro¬ 
cesses  and  control  methods  to  minimize  risk  in 
both  Government  and  industry.  The  task  force 
defined  these  critical  events  in  terms  of  the  tem¬ 
plates,  which  are  briefly  discussed  later.  A  copy 
of  DoD  4245. 7-M  may  be  obtained  at  the  De¬ 
fense  Technical  Information  Center  (DTIC) 
Web  site :  http  ://www.dtic.mil/whs/directives . 

Additional  areas,  such  as  manpower,  environ¬ 
mental  impact,  systems  safety  and  health,  and 
systems  engineering,  that  are  analyzed  during 
program  plan  development  provide  indicators 
for  additional  risk.  The  PMO  should  consider 
these  areas  for  early  assessment  since  failure 
to  do  so  could  cause  dire  consequences/impacts 
in  the  program’s  latter  phases. 
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Figure  2-2.  Critical  Process  Areas  and  Templates 


In  addition,  PMs  should  address  the  uncertainty 
associated  with  security  —  an  area  sometimes 
overlooked  by  developers  but  addressed  under 
the  topic  of  acquisition  system  protection  in  the 
Defense  Acquisition  Deskbook.  However,  in  ad¬ 
dition  to  the  guidance  given  there,  PMs  must  rec¬ 
ognize  that,  in  the  past,  classified  programs  have 
experienced  difficulty  in  access,  facilities,  clear¬ 
ances,  and  visitor  control.  Failure  to  manage  these 
aspects  of  a  classified  program  could  adversely 
affect  cost  and  schedule. 

2.5  RISK  PLANNING 

2.5.1  Purpose  of  Risk  Plans 

Risk  planning  is  the  detailed  formulation  of  a 
program  of  action  for  the  management  of  risk.  It 
is  the  process  to: 


•  Develop  and  document  an  organized,  com¬ 
prehensive,  and  interactive  risk  management 
strategy. 

•  Determine  the  methods  to  be  used  to  execute 
a  PM’s  risk  management  strategy. 

•  Plan  for  adequate  resources. 

Risk  planning  is  iterative  and  includes  describ¬ 
ing  and  scheduling  the  activities  and  process  to 
assess  (identify  and  analyze),  handle,  monitor, 
and  document  the  risk  associated  with  a  pro¬ 
gram.  The  result  is  the  Risk  Management  Plan 
(RMP). 

2.5.2  Risk  Planning  Process 

The  PMO  should  periodically  review  the  plan  and 
revise  it,  if  necessary.  Some  events  such  as:  (1)  a 
change  in  acquisition  strategy,  (2)  preparation  for 
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a  major  decision  point,  (3)  technical  audits  and 
reviews,  (4)  an  update  of  other  program  plans, 
and  (5)  preparation  for  a  Program  Objective 
Memorandum  (POM)  submission  may  drive  the 
need  to  update  an  existing  plan. 

Planning  begins  by  developing  and  document¬ 
ing  a  risk  management  strategy.  Early  efforts 
establish  the  purpose  and  objective,  assign  re¬ 
sponsibilities  for  specific  areas,  identify  addi¬ 
tional  technical  expertise  needed,  describe  the 
assessment  process  and  areas  to  consider, 
delineate  procedures  for  consideration  of  han¬ 
dling  options,  define  a  risk  rating  scheme, 
dictate  the  reporting  and  documentation  needs, 
and  establish  report  requirements  and  moni¬ 
toring  metrics.  This  planning  should  also  ad¬ 
dress  evaluation  of  the  capabilities  of  potential 
sources  as  well  as  early  industry  involvement  and 
program. 

The  PM’s  strategy  to  manage  risk  provides  the 
program  team  with  direction  and  basis  for  plan¬ 
ning.  Initially  formalized  during  a  program’s 
Concept  Exploration  Phase  and  updated  for 
each  subsequent  program  phase,  the  strategy 
should  be  reflected  in  the  program’s  acquisi¬ 
tion  strategy,  which  with  requirement  and  threat 
documents,  known  risks,  and  system  and  pro¬ 
gram  characteristics  are  sources  of  information 
for  PMO  use  to  devise  a  strategy  and  begin  de¬ 


veloping  a  Risk  Management  Plan.  Since  the 
program’s  risks  are  affected  by  the  Government 
and  contractor  team’s  ability  to  develop  and 
manufacture  the  system,  industry  can  provide 
valuable  insight  into  this  area  of  consideration. 

The  plan  is  the  road  map  that  tells  the  Govern¬ 
ment  and  contractor  team  how  to  get  from  where 
the  program  is  today  to  where  the  PM  wants  it 
to  be  in  the  future.  The  key  to  writing  a  good 
plan  is  to  provide  the  necessary  information  so 
the  program  team  knows  the  objectives,  goals, 
and  the  PMO’s  risk  management  process.  Since 
it  is  a  map,  it  may  be  specific  in  some  areas, 
such  as  the  assignment  of  responsibilities  for 
Government  and  contractor  participants  and 
definitions,  and  general  in  other  areas  to  allow 
users  to  choose  the  most  efficient  way  to  pro¬ 
ceed.  For  example,  a  description  of  techniques 
that  suggests  several  methods  for  evaluators  to 
use  to  assess  risk  is  appropriate,  since  every 
technique  has  advantages  and  disadvantages 
depending  on  the  situation. 

Appendix  B  contains  two  examples  of  a  risk 
plan  and  a  summary  of  the  format  is  shown  in 
Figure  2-3. 

In  a  decentralized  PMO  risk  management  orga¬ 
nization,  the  program’s  risk  management  coor¬ 
dinator  may  be  responsible  for  risk  management 
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Figure  2-3.  A  Risk  Management  Plan  Outline/Format 
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planning.  See  Sections  4.4,  Risk  Management 
Organization  in  the  PMO,  and  5.3,  Risk  Plan¬ 
ning  Techniques. 

2.6  RISK  ASSESSMENT 

2.6.1  Purpose  of  Risk  Assessments 

The  primary  objective  of  assessments  is  to 
identify  and  analyze  program  risks  so  that  the 
most  critical  among  them  may  be  controlled. 
Assessments  are  factors  that  managers  should 
consider  in  setting  cost,  schedule,  and  perfor¬ 
mance  objectives  because  they  provide  an 
indication  of  the  probability/likelihood  of 
achieving  the  desired  outcomes. 

2.6.2  Risk  Assessment  Process 

Risk  assessment  is  the  problem  definition  stage 
of  management  that  identifies  and  analyzes 
(quantifies)  prospective  program  events  in  terms 
of  probability  and  consequences/impacts.  The 
results  form  the  basis  for  most  risk  management 
actions.  It  is  probably  the  most  difficult  and 
time-consuming  part  of  the  management  pro¬ 
cess.  There  are  no  quick  answers  or  shortcuts. 
Tools  are  available  to  assist  evaluators  in  assess¬ 
ing  risk,  but  none  are  totally  suitable  for  any 
program  and  may  be  highly  misleading  if  the 
user  does  not  understand  how  to  apply  them  or 
interpret  the  results.  Despite  its  complexity,  risk 
assessment  is  one  of  the  most  important  phases 
of  the  risk  process  because  the  caliber  and  quality 
of  assessments  determine  the  effectiveness  of  a 
management  program. 

The  components  of  assessment,  identification 
and  analysis,  are  performed  sequentially  with 
identification  being  the  first  step. 

Risk  identification  begins  by  compiling  the 
program’s  risk  events.  PMOs  should  examine 
and  identify  program  events  by  reducing  them 
to  a  level  of  detail  that  permits  an  evaluator  to 


understand  the  significance  of  any  risk  and  iden¬ 
tify  its  causes,  i.e.,  risk  drivers.  This  is  a  practi¬ 
cal  way  of  addressing  the  large  and  diverse  num¬ 
ber  of  potential  risks  that  often  occur  in  acqui¬ 
sition  programs.  For  example,  a  WBS  level  4 
or  5  element  may  generate  several  risk  events 
associated  with  a  specification  or  function,  e.g., 
failure  to  meet  turbine  blade  vibration  require¬ 
ments  for  an  engine  turbine  design. 

Risk  events  are  best  identified  by  examining 
each  WBS  product  and  process  element  in  terms 
of  the  sources  or  areas  of  risk,  as  previously 
described  in  Paragraph  2.4.2. 

Risks  are  those  events  that  evaluators  (after 
examining  scenarios,  WBS,  or  processes) 
determine  would  adversely  affect  the  program. 
Evaluators  may  initially  rank  events  by  prob¬ 
ability  and  consequence/impact  of  occurrence 
before  beginning  analysis  to  focus  on  those 
most  critical. 

Risk  analysis  is  a  technical  and  systematic  pro¬ 
cess  to  examine  identified  risks,  isolate  causes, 
determine  the  relationship  to  other  risks,  and 
express  the  impact  in  terms  of  probability  and 
consequences/impacts. 

In  practice,  the  distinction  between  risk  identifi¬ 
cation  and  risk  analysis  is  often  blurred  because 
there  is  some  risk  analysis  that  occurs  during  the 
identification  process.  For  example,  if,  in  the  pro¬ 
cess  of  interviewing  an  expert,  a  risk  is  identified, 
it  is  logical  to  pursue  informa-tion  on  the  prob¬ 
ability  of  it  occurring,  the  consequences/impacts, 
the  time  associated  with  the  risk  (i.e.,  when  it  might 
occur),  and  possible  ways  of  dealing  with  it.  The 
latter  actions  are  part  of  risk  analysis  and  risk  han¬ 
dling,  but  often  begin  during  risk  identification. 

Prioritization  is  the  ranking  of  risk  events  to  de¬ 
termine  the  order  of  importance.  It  serves  as  the 
basis  for  risk-handling  actions.  Prioritization  is 
part  of  risk  analysis. 
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Integrated  Product  Teams  (IPTs)  typically  per¬ 
form  risk  assessments  in  a  decentralized  risk  man¬ 
agement  organization  as  described  in  Paragraph 
4.4.  If  necessary,  the  team  may  be  augmented 
by  people  from  other  program  areas  or  outside 
experts.  Paragraph  5.4,  Risk  Assessment  Tech¬ 
niques,  elaborates  on  this  for  each  of  the  de¬ 
scribed  assessment  techniques. 


2.6.3  Timing  of  Risk  Assessments 

The  assessment  process  begins  during  the 
Concept  Refinement  (CR)  Phase  and  continues 
throughout  the  subsequent  acquisition  phases. 
The  PMO  should  continually  reassess  the  pro¬ 
gram  at  increasing  levels  of  detail  as  the  pro¬ 
gram  progresses  through  the  acquisition 
phases  and  more  information  becomes  avail¬ 
able.  There  are,  however,  times  when  events 


Figure  2-4.  Risk  Assessment 
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may  require  new  assessments,  i.e.,  a  major 
change  in  the  acquisition  strategy.  Paragraph 
2.5.2  lists  other  events  that  could  cause  risk 
assessments  to  be  performed. 

2.6.4  Conducting  Risk  Assessments 

There  is  no  standard  approach  to  assessing  risk 
because  methods  vary  according  to  the  tech¬ 
nique  employed,  the  phase  of  the  program,  and 
the  nature  of  the  program  itself;  however,  some 
top-level  actions  are  typically  common  to  all 
methods.  They  are  grouped  in  Figure  2-4  into 
pre-risk  assessment  activities,  risk  identifica¬ 
tion  activities,  and  risk  analysis  activities.  Each 
risk  category  or  area,  e.g.,  cost,  schedule,  and 
performance,  includes  a  core  set  of  assessment 
tasks  and  is  related  to  the  other  two  categories. 
This  relationship  requires  supportive  analysis 
among  areas  to  ensure  the  integration  of  the  as¬ 
sessment  process.  For  example,  a  technical  as¬ 
sessment  probably  should  include  a  cost  and 
schedule  analysis  in  determining  the  technical 
risk  impact.  The  results  of  the  assessments,  nor¬ 
mally  conducted  by  IPTs  follow: 

Performance/Technical  Assessment  (Includes 
technical  areas  of  risk  shown  in  Paragraph 
2.4.2.) 

•  Provides  technical  foundation, 

•  Identifies  and  describes  program  risks,  i.e., 
threat,  technology,  design,  manufacturing, 
etc., 

•  Prioritizes  risks  with  relative  or  quantified 
weight  for  program  impact, 

•  Analyzes  risks  and  relates  them  to  other 
internal  and  external  risks, 

•  Quantifies  associated  program  activities  with 
both  time  duration  and  resources, 


•  Quantifies  inputs  for  schedule  assessment 
and  cost  estimate, 

•  Documents  technical  basis  and  risk  definition 
for  the  risk  assessment. 

Schedule  Assessment 

•  Evaluates  baseline  schedule  inputs, 

•  Incorporates  technical  assessment  and 
schedule  uncertainty  inputs  to  program 
schedule  model, 

•  Evaluates  impacts  to  program  schedule 
based  on  technical  team  assessment, 

•  Performs  schedule  analysis  on  program 
integrated  master  schedule, 

•  Quantifies  schedule  excursions  reflecting 
effects  of  cost  risks,  including  resource 
constraints, 

•  Provides  Government  schedule  assessment 
for  cost  analysis  and  fiscal  year  planning, 

•  Reflects  technical  foundation,  activity  defi¬ 
nition,  and  inputs  from  technical  and  cost  ar¬ 
eas, 

•  Documents  schedule  basis  and  risk  impacts 
for  the  risk  assessment. 

Cost  Estimate  and  Assessment 

•  Builds  on  technical  and  schedule  assessment 
results, 

•  Translates  technical  and  schedule  risks  into 
cost, 

•  Derives  cost  estimate  by  integrating  techni¬ 
cal  risk  and  schedule  risk  impacts  with 
resources, 
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•  Establishes  budgetary  requirements  consistent 
with  fiscal  year  planning, 

•  Determines  if  the  phasing  of  funds  supports 
technical  and  acquisition  approach, 

•  Provides  program  cost  excursions  from: 

-  Near-term  budget  execution  impacts, 

-  External  budget  changes  and  constraints. 

•  Documents  cost  basis  and  risk  impacts. 

2.6.4.1  Pre-Risk  Assessment  Activities.  The  Risk 
Management  Plan  may  describe  the  actions  that 
compose  this  activity.  Typically,  a  program-level 
IPT  may  conduct  a  quick-look  assessment  of  the 
program  to  identify  the  need  for  technical  experts 
(who  are  not  part  of  the  team)  and  to  examine  ar¬ 
eas  that  appear  most  likely  to  contain  risk.  The 
program’s  risk  coordinator,  or  an  outside  expert, 
may  train  the  IPTs,  focusing  on  the  program’s  risk 
strategy,  definitions,  suggested  techniques,  docu¬ 
mentation,  and  reporting  requirements.  Paragraph 
4.9,  Risk  Management  Training,  provides  some  sug¬ 
gestions  for  training. 

2.6.4.2  Risk  Identification  Activity.  To  iden¬ 
tify  risk  events,  IPTs  should  break  down  pro¬ 
gram  elements  to  a  level  where  they,  or  subject- 
matter  experts,  can  perform  valid  assessments. 


The  information  necessary  to  do  this  varies  ac¬ 
cording  to  the  phase  of  the  program.  During  the 
early  phases,  requirement,  threat  documents,  and 
acquisition  plans  may  be  the  only  program-spe¬ 
cific  data  available.  They  should  be  analyzed  to 
identify  events  that  may  have  adverse  conse¬ 
quences/impacts.  A  useful  initial  identification 
exercise  is  to  perform  a  mission  profile  for  the 
system  as  suggested  in  DoD  4245 .7-M,  Transi¬ 
tion  from  Development  to  Production.  Using  this 
methodology,  the  developer  creates  a  functional 
and  environmental  profile  for  the  system  and  ex¬ 
amines  the  low-level  requirements  that  the  sys¬ 
tem  must  meet  to  satisfy  its  mission  requirements. 
The  IPTs  may  then  study  these  requirements  to 
determine  which  are  critical.  For  example,  in  an 
aircraft  profile,  it  may  be  apparent  that  high  speed 
is  critical.  If  the  speed  requirement  is  close  to 
that  achieved  by  existing  aircraft,  this  may  not 
be  a  concern.  However,  if  the  speed  is  greater 
than  that  achieved  by  today’s  aircraft,  it  may  be 
a  critical  risk  area.  Since  aircraft  speed  depends, 
among  other  things,  on  weight  and  engine  thrust, 
it  would  be  desirable  to  enhst  the  help  of  a  mate¬ 
rials  expert  to  address  weight  and  an  engine  ex¬ 
pert  to  assess  engine-associated  risk. 

Another  method  of  decomposition  is  to  create  a 
WBS  as  early  as  possible  in  a  program.  Figure 
2-5  is  a  simple  example  of  a  decomposition  based 
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on  the  WBS  for  an  aircraft.  The  figure  shows  an 
important  requirement  of  the  decomposition  pro¬ 
cess,  the  establishment  of  goals  (e.g.,  don’t  ex¬ 
ceed  the  weight  budget  or  objective).  Risk  events 
are  determined  by  matching  each  WBS  element 
and  process  to  sources  or  areas  of  risk.  Risk  ar¬ 
eas/sources  are  described  in  Paragraph  2.4.2  and 
Table  4-2. 

During  decomposition,  risk  events  are  identified 
from  experience,  brainstorming,  lessons  learned 
from  similar  programs,  and  guidance  contained 
in  the  risk  management  plan.  A  structured  ap¬ 
proach  previously  discussed  matches  each  WBS 
element  and  process  in  terms  of  sources  or  areas 
of  risk.  The  examination  of  each  element  against 
each  risk  area  is  an  exploratory  exercise  to  identify 
the  critical  risks.  The  investigation  may  show  that 
risks  are  interrelated.  For  example,  the  weight  of 
an  aircraft  affects  its  speed,  but  also  impacts  the 
payload,  range,  and  fuel  requirements.  These  have 
design  and  logistics  consequences/impacts  and 
may  even  affect  the  number  of  aircraft  that  must 
be  procured  to  meet  objectives. 

Critical  risks  need  to  be  documented  as  speci¬ 
fied  in  the  Risk  Management  Plan  and  may  in¬ 
clude  the  scenario  that  causes  the  risk,  planned 
management  controls  and  actions,  etc.  It  may  also 
contain  an  initial  assessment  of  the  conse¬ 
quences/impacts  to  focus  the  risk  assessment  ef¬ 
fort.  A  risk  watch  list  should  be  initiated  as  part 
of  risk  identification.  It  is  refined  during  han¬ 
dling,  and  monitored/updated  during  the  moni¬ 
toring  phase.  Watch  lists  provide  a  convenient 
and  necessary  form  to  track  and  document  ac¬ 
tivities  and  actions  resulting  from  risk  analysis. 
Watch  lists  frequently  evolve  from  the  input  of 
each  “expert”  functional  manager  on  a  program. 
(See  paragraph  5.7.5.) 

2.6.4.3  Risk  Analysis  Activity.  Analysis  begins 
with  a  detailed  study  of  the  critical  risk  events 
that  have  been  identified.  The  objective  is  to 
gather  enough  information  about  the  risks  to 


judge  the  probability  of  occurrence  and  the  im¬ 
pact  on  cost,  schedule,  and  performance  if  the 
risk  occurs. 

Impact  assessments  are  normally  subjective  and 
based  on  detailed  information  that  may  come 
from: 

•  Comparisons  with  similar  systems, 

•  Relevant  lessons-learned  studies, 

•  Experience, 

•  Results  from  tests  and  prototype  development, 

•  Data  from  engineering  or  other  models, 

•  Specialist  and  expert  judgments, 

•  Analysis  of  plans  and  related  documents, 

•  Modeling  and  simulation, 

•  Sensitivity  analysis  of  alternatives. 

Depending  on  the  particular  technique  and  the 
risk  being  analyzed,  some  supporting  analysis 
may  be  necessary,  i.e.,  analysis  of  contractor  pro¬ 
cesses,  such  as  design,  engineering,  fault  tree 
analysis,  engineering  models,  simulation,  etc. 
Analyses  provide  the  basis  for  subjective 
assessments. 

A  critical  aspect  of  risk  analysis  is  data 
collection.  Two  primary  sources  of  data  are 
interviews  of  subject-matter  experts  and  analogy 
comparisons  with  similar  systems.  Paragraph  5.4 
contains  a  procedure  for  collecting  both  types 
of  data  for  use  in  support  of  the  techniques  listed 
in  Table  2-1.  Periodically,  sets  of  risks  need  to 
be  prioritized  in  preparation  for  risk  handling, 
and  aggregated  to  support  program  management 
reviews.  Paragraph  5.5,  Risk  Prioritization, 
describes  methods  for  accomplishing  this. 
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2.6.4.3.1  Risk  Rating  and  Prioritization/ 
Ranking 

Risk  ratings  are  an  indication  of  the  potential  im¬ 
pact  of  risks  on  a  program;  they  are  a  measure 
of  the  probability/likelihood  of  an  event 
occurring  and  the  consequences/impacts  of  the 
event.  They  are  often  expressed  as  high, 


moderate,  and  low.  Risk  rating  and  prioritiza¬ 
tion/ranking  are  considered  integral  parts  of  risk 
analysis. 

A  group  of  experts,  who  are  familiar  with  each 
risk  source/area  (e.g.,  design,  logistics,  produc¬ 
tion,  etc.)  and  product  WBS  element,  are  best 
qualified  to  determine  risk  ratings.  They  should 


Risk  Assessment  Technique 

Applicable  Acquisition  Phases 

Applicable  Risk  Areas  & 
Processes 

Plan  Evaluation/Risk  Identification 

All  phases 

Program  Plans  and  critical  com¬ 
munications  with  the  developer 

Product  (WBS)  Risk  Assessment 

All  phases  starting  with  the 
completion  of  the  Contract  WBS 

All  critical  risk  areas  except  threat, 
requirements,  cost,  and  schedule 

Process  (DoD  4265. 7-M)  Risk 
Assessment 

All  phases,  but  mainly  late  SDD 

All  critical  risk  processes 

Cost  Risk  Assessment 

All  phases 

Cost  critical  risk  areas 

Schedule  Risk  Assessment 

All  phases 

Schedule  critical  risk  areas 

Table  2-1.  Risk  Assessment  Approaches 


Level 

What  is  the  Likelihood  the  Risk 

Event  Will  Happen? 

a 

Remote 

b 

Unlikely 

c 

Likely 

d 

Highly  Likely 

e 

Near  Certainty 

Table  2-2.  Probability/Likelihood  Criteria  (Example) 


Level 

Given  the  Risk  Is  Realized,  What  Is  the  Magnitude  of  the  Impact? 

Performance 

Schedule 

Cost 

a 

Minimal  or  no  impact 

Minimal  or  no  impact 

Minimal  or  no  impact 

b 

Acceptable  with  some 
reduction  in  margin 

Additional  resources 
required;  able  to  meet 
need  dates 

<5% 

c 

Acceptable  with  significant 
reduction  in  margin 

Minor  slip  in  key  milestones; 
not  able  to  meet  need  date 

5-7% 

d 

Acceptable;  no  remaining 
margin 

Major  slip  in  key  milestone 
or  critical  path  impacted 

7-10% 

e 

Unacceptable 

Can’t  achieve  key  team  or 
major  program  milestone 

>10% 

Table  2-3.  Consequences/Impacts  Criteria  (Example) 
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Risk  Rating 

Description 

High 

Major  disruption  likely 

Moderate 

Some  disruption 

Low 

Minimum  disruption 

Table  2-4.  Overall  Risk  Rating  Criteria  (Example) 


Priority 

Area/Source 

Process 

Location 

Risk  Event 

Proba¬ 

bility 

Conse¬ 

quence 

Risk 

Rating 

1 

Design 

WBS3.1 

Design  not 
completed  on  time 

Highly 

Likely 

Can’t  achieve 
key  milestone 

High 

2 

3 

Table  2-5.  Risk  Ratings  (Example) 


identify  rating  criteria  for  review  by  the  PMO, 
who  includes  them  in  the  Risk  Management 
Plan.  In  most  cases,  the  criteria  will  be  based 
on  the  experience  of  the  experts,  as  opposed  to 
mathematically  derived,  and  should  establish 
levels  of  probability/likelihood  and  conse¬ 
quences/  impacts  that  will  provide  a  range  of 
possibilities  large  enough  to  distinguish  differ¬ 
ences  in  risk  ratings.  At  the  program  level,  con¬ 
sequences/impacts  should  be  expressed  in 
terms  of  impact  on  cost,  schedule  and  perfor¬ 
mance.  Tables  2-2  and  2-3  are  examples  of 
probability/  likelihood  and  consequence/impact 
criteria,  and  Table  2-4  contains  an  example  of 
overall  risk  rating  criteria,  which  considers  both 
probability/likelihood  and  consequences/ 
impacts.  Table  2-5  provides  a  sample  format 
for  presenting  risk  ratings. 

Using  these  risk  ratings,  PMs  can  identify 
events  requiring  priority  management  (high  or 
moderate  risk  probability /likelihood  or  conse¬ 
quences/impacts).  The  document  prioritizing 
the  risk  events  is  called  a  Watch  List.  Risk  rat¬ 
ings  also  help  to  identify  the  areas  that  should 
be  reported  within  and  outside  the  PMO,  e.g., 
milestone  decision  reviews.  Thus,  it  is  impor¬ 
tant  that  the  ratings  be  portrayed  as  accurately 
as  possible. 


A  simple  method  of  representing  the  risk  rating 
for  risk  events,  i.e.,  a  risk  matrix,  is  shown  in 
Figure  2-6.  In  this  example  matrix,  the  PM  has 
defined  high,  moderate,  and  low  levels  for  the 
various  combinations  of  probability  /likelihood 
and  consequences/impacts.  The  matrix  is  struc¬ 
tured  somewhat  symmetrically;  programs  should 
tailor  the  scales  and  risk  rating  blocks  to  match 
their  unique  risk  management  requirements. 

There  is  a  common  tendency  to  attempt  to  de¬ 
velop  a  single  number  to  portray  the  risk  associ¬ 
ated  with  a  particular  event.  This  approach  may 
be  suitable  if  both  probability/likelihood  (prob¬ 
ability)  and  consequences/impacts  have  been 
quantified  using  compatible  cardinal  scales  or 
calibrated  ordinal  scales  whose  scale  levels  have 
been  determined  using  accepted  procedures  (e.g., 
Analytical  Hierarchy  Process).  In  such  a  case, 
mathematical  manipulation  of  the  values  may  be 
meaningful  and  provide  some  quantitative  basis 
for  the  ranking  of  risks. 

In  most  cases,  however,  risk  scales  are  actually 
just  raw  (uncalibrated)  ordinal  scales,  reflect¬ 
ing  only  relative  standing  between  scale  levels 
and  not  actual  numerical  differences.  Any  math¬ 
ematical  operations  performed  on  results  from 
uncalibrated  ordinal  scales,  or  a  combination 
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of  uncalibrated  ordinal  and  cardinal  scales,  can 
provide  information  that  will  at  best  be  mislead¬ 
ing,  if  not  completely  meaningless,  resulting  in 
erroneous  risk  ratings.  Hence,  mathematical  op¬ 
erations  should  generally  not  be  performed  on 
scores  derived  from  uncalibrated  ordinal  scales. 
(Note:  risk  scales  that  are  expressed  as  decimal 
values  (e.g.,  a  5  level  scale  with  values  0.2, 0.4, 
0.6, 0.8  and  1 .0)  still  retain  the  ordinal  scale  limi¬ 
tations  discussed  above.)  For  a  more  detailed  dis¬ 
cussion  of  risk  scales,  see  Appendix  G  of  the 
reference  Effective  Risk  Management:  Some 
Keys  to  Success. 

One  way  to  avoid  this  situation  is  to  simply  show 
each  risk  event’s  probability /likelihood  and  con¬ 
sequences/impacts  separately,  with  no  attempt 
to  mathematically  combine  them.  Other  factors 
that  may  significantly  contribute  to  the  risk  rat¬ 
ing,  such  as  time  sensitivity  or  resource  avail¬ 
ability,  can  also  be  shown.  The  prioritization  or 
ranking  —  done  after  the  rating  —  should  also 
be  performed  using  a  structured  risk  rating  ap¬ 
proach  (e.g.,  Figure  2-6)  coupled  with  expert 
opinion  and  experience.  Prioritization  or  rank¬ 
ing  is  achieved  through  integration  of  risk  events 
from  lower  to  higher  WBS  levels.  This  means 
that  the  effect  of  risk  at  lower  WBS  elements 
needs  to  be  reflected  cumulatively  at  the  top  or 
system  level. 


2.7  RISK  HANDLING 

2.7.1  Purpose  of  Risk  Handling 

Risk  handling  includes  specific  methods  and 
techniques  to  deal  with  known  risks  and  a 
schedule  for  accomplishing  tasks,  identifies  who 
is  responsible  for  the  risk  area,  and  provides  an 
estimate  of  the  cost  and  schedule  associated  with 
handling  the  risk,  if  any.  It  involves  planning  and 
execution  with  the  objective  of  handling  risks  at 
an  acceptable  level.  The  IPTs  that  assess  risk 
should  begin  the  process  to  identify  and  evaluate 
handling  approaches  to  propose  to  the  PM,  who 
selects  the  appropriate  ones  for  implementation. 

2.7.2  Risk-Handling  Process 

The  risk-handling  phase  must  be  compatible  with 
the  risk  management  plan  and  any  additional 
guidance  the  PM  provides.  Paragraph  5.3  de¬ 
scribes  a  technique  that  concentrates  on  plan¬ 
ning.  A  critical  part  of  planning  involves  refin¬ 
ing  and  selecting  of  the  most  appropriate  han¬ 
dling  options. 

The  IPTs  that  evaluate  the  handling  options  may 
use  the  following  criteria  as  a  starting  point  for 
assessment: 

•  Can  the  option  be  feasibly  implemented  and 
still  meet  the  user’s  needs? 


u 

o 

o 


a> 


e 

M 

M 

H 

H 

H 

d 

L 

M 

M 

H 

H 

c 

L 

L 

M 

M 

H 

b 

L 

L 

L 

M 

M 

a 

L 

L 

L 

L 

M 

a 

b 

c 

d 

e 

Consequence 


Figure  2-6.  Overall  Risk  Rating  (Example) 
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•  What  is  the  expected  effectiveness  of  the  han¬ 
dling  option  in  reducing  program  risk  to  an 
acceptable  level? 

•  Is  the  option  affordable  in  terms  of  dollars  and 
other  resources  (e.g.,  use  of  critical  materials, 
test  facilities,  etc.)? 

•  Is  time  available  to  develop  and  implement 
the  option,  and  what  effect  does  that  have 
on  the  overall  program  schedule? 

•  What  effect  does  the  option  have  on  the 
system’s  technical  performance? 

Risk-handling  options  can  include  risk  control, 
risk  avoidance,  risk  assumption,  and  risk 
transfer.  An  acronym  used  to  identify  these  op¬ 
tions  is  “CAAT.”  Although  the  control  risk¬ 
handling  option  is  commonly  used  in  defense 
programs,  it  should  not  automatically  be  cho¬ 
sen.  All  four  options  should  be  evaluated  and 
the  best  one  chosen  for  a  given  risk  issue. 

Risk  Control  does  not  attempt  to  eliminate  the 
source  of  the  risk  but  seeks  to  reduce  or  mitigate 
the  risks.  It  monitors  and  manages  the  risk  in  a 
manner  that  reduces  the  probability  /likelihood 
and/or  consequence/impact  of  its  occurrence  or 
minimizes  the  risk’s  effect  on  the  program.  This 
option  may  add  to  the  cost  of  a  program;  how¬ 
ever,  the  selected  approach  should  provide  an 
optional  risk  among  the  candidate  approaches 
of  risk  reduction,  cost  effectiveness,  and  sched¬ 
ule  impact.  A  sampling  is  listed  below  of  the 
types  of  risk  control  actions  available  to  the 
PMO.  Paragraph  5.6.2  discusses  them  in  more 
detail. 

•  Multiple  Development  Efforts.  Create 
competing  systems  in  parallel  that  meet  the 
same  performance  requirements. 

•  Alternative  Design.  Create  a  backup  design 
option  that  uses  a  lower  risk  approach. 


•  Trade  Studies.  Arrive  at  a  balance  of  engi¬ 
neering  requirements  in  the  design  of  a 
system. 

•  Early  Prototyping.  Build  and  test  prototypes 
early  in  the  system  development. 

•  Incremental  Development.  Design  with  the 
intent  of  upgrading  system  parts  in  the  future. 

•  Technology  Maturation  Efforts .  Normally, 
technology  maturation  is  used  when  the  de¬ 
sired  technology  will  replace  an  existing 
technology  which  is  available  for  use  in  the 
system. 

•  Robust  Design.  This  approach,  while  it  could 
be  more  costly,  uses  advanced  design  and 
manufacturing  techniques  that  promote  qual¬ 
ity  through  design. 

•  Reviews,  Walk-throughs,  and  Inspections. 

These  three  actions  can  be  used  to  reduce  the 
probability/likelihood  and  potential  conse¬ 
quences/impacts  of  risks  through  timely  as¬ 
sessment  of  actual  or  planned  events. 

•  Design  of  Experiments.  This  engineering 
tool  identifies  critical  design  factors  that  are 
sensitive,  therefore  potentially  high  risk,  to 
achieve  a  particular  user  requirement. 

•  Open  Systems.  Carefully  selected  commer¬ 
cial  specifications  and  standards  whose  use 
can  result  in  lower  risks. 

•  Use  of  Standard  Items/Software  Reuse. 

Use  of  existing  and  proven  hardware  and 
software,  where  applicable,  can  substantially 
reduce  risks. 

•  Two-Phase  Development.  Incorporation  of 
formal  risk  reduction  into  System  Develop¬ 
ment  and  Demonstration  (SDD).  The  first 
part  of  SDD  is  System  Integration  (SI),  where 
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prototypes  are  developed  and  tested.  In  the 
second  part,  System  Demonstration  (SD), 
Engineering  Development  Models  (EDMs) 
are  developed  and  tested. 

•  Use  of  Mock-ups.  The  use  of  mock-ups, 
especially  man-machine  interface  mock-ups, 
can  be  used  to  conduct  early  exploration  of 
design  options. 

•  Modeling/Simulation.  Modeling  and  simu¬ 
lation  can  be  used  to  investigate  various  de¬ 
sign  options  and  system  requirement  levels. 

•  Key  Parameter  Control  Boards .  The  prac¬ 
tice  of  establishing  a  control  board  for  a 
parameter  may  be  appropriate  when  a  par¬ 
ticular  feature  (such  as  system  weight)  is 
crucial  to  achieving  the  overall  program 
requirements. 

•  Manufacturing  Screening.  For  programs 
in  SDD,  various  manufacturing  screens 
(including  environmental  stress  screening 
(ESS))  can  be  incorporated  into  test  article 
production  and  low  rate  initial  production 
(LRIP)  to  identify  deficient  manufactur¬ 
ing  processes.  ESS  is  a  manufacturing  pro¬ 
cess  for  stimulating  parts  and  workman¬ 
ship  defects  in  electronic  assemblies  and 
units. 

•  Test,  Analyze,  and  Fix  (TAAF).  TAAF  is 
the  use  of  a  period  of  dedicated  testing  to 
identify  and  correct  deficiencies  in  a  design. 

•  Demonstration  Events.  Demonstration 
events  are  points  in  the  program  (normally 
tests)  that  determine  if  risks  are  being 
successfully  abated. 

•  Process  Proofing.  Similar  to  Program  Met¬ 
rics,  but  aimed  at  manufacturing  and  support 
processes  which  are  critical  to  achieving  sys¬ 
tem  requirements.  Proofing  simulates  actual 


production  environments  and  conditions  to 
insure  repeatedly  conforming  hardware  and 
software. 

As  you  can  see,  there  are  numerous  means  that 
can  be  used  to  actively  control  risks. 

Risk  Avoidance  involves  a  change  in  the  con¬ 
cept,  requirements,  specifications,  and/or  prac¬ 
tices  that  reduce  risk  to  an  acceptable  level.  Sim¬ 
ply  stated,  it  eliminates  the  sources  of  high  or 
possibly  medium  risk  and  replaces  them  with  a 
lower  risk  solution  and  may  be  supported  by  a 
cost/benefit  analysis.  Generally,  this  method  may 
be  done  in  parallel  with  the  up-front 
requirements  analysis,  supported  by  cost/ 
requirement  trade  studies,  which  can  include 
Cost  As  an  Independent  Variable  (CAIV)  trades. 

Risk  Assumption.  Risk  assumption  is  an 
acknowledgment  of  the  existence  of  a  particu¬ 
lar  risk  situation  and  a  conscious  decision  to 
accept  the  associated  level  of  risk,  without 
engaging  in  any  special  efforts  to  control  it. 
However,  a  general  cost  and  schedule  reserve 
may  be  set  aside  to  deal  with  any  problems 
that  may  occur  as  a  result  of  various  risk  as¬ 
sumption  decisions.  This  method  recognizes 
that  not  all  identified  program  risks  warrant 
special  handling;  as  such,  it  is  most  suited  for 
those  situations  that  have  been  classified  as 
low  risk.  The  key  to  successful  risk  assump¬ 
tion  is  twofold: 

•  Identify  the  resources  (time,  money,  people, 
etc.)  needed  to  overcome  a  risk  if  it  materi¬ 
alizes.  This  includes  identifying  the  specific 
management  actions  (such  as  retesting, 
additional  time  for  further  design  activities) 
that  may  occur. 

•  Ensure  that  necessary  administrative  actions 
are  taken  to  identify  a  management  reserve 
to  accomplish  those  management  actions. 
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Risk-handling  options  have  broad  cost  impli¬ 
cations.  The  magnitude  of  these  costs  are  cir¬ 
cumstance-dependent.  The  approval  and  fund¬ 
ing  of  handling  options  should  be  part  of  the 
process  that  establishes  the  program  cost  and  per¬ 
formance  goals.  This  should  normally  be  done 
by  the  Program-Level  Risk  Management  IPT  or 
Risk  Management  Board.  The  selected  handling 
option  should  be  included  in  the  program’s  ac¬ 
quisition  strategy. 

Once  the  acquisition  strategy  includes  risk¬ 
handling  approaches,  the  PMO  can  derive  the 
schedule  and  identify  cost,  schedule,  and 
performance,  impacts  to  the  basic  program. 

Risk  Transfer.  This  action  may  reallocate  risk 
during  the  concept  development  and  design  pro¬ 
cesses  from  one  part  of  the  system  to  another, 
thereby  reducing  the  overall  system  risk,  or  re¬ 
distributing  risks  between  the  Government  and 
the  prime  contractor  or  within  Government 
agencies;  or  between  members  of  the  contrac¬ 
tor  team.  It  is  an  integral  part  of  the  functional 
analysis  process.  Risk  transfer  is  a  form  of  risk 
sharing  and  not  risk  abrogation  on  the  part  of 
the  Government,  and  it  may  influence  cost  ob¬ 
jectives.  An  example  is  the  transfer  of  a  func¬ 
tion  from  hardware  implementation  to  software 
implementation  or  vice  versa.  The  effectiveness 
of  risk  transfer  depends  on  the  use  of  success¬ 
ful  system  design  techniques.  Modularity  and 
functional  partitioning  are  two  design  tech¬ 
niques  that  support  risk  transfer.  In  some  cases, 
risk  transfer  may  concentrate  risk  areas  in  one 
area  of  the  design.  This  allows  management  to 
focus  attention  and  resources  on  that  area. 

2.8  RISK  MONITORING 

The  monitoring  process  systematically  tracks 
and  evaluates  the  effectiveness  of  risk-han¬ 
dling  actions  against  established  metrics. 
Monitoring  results  may  also  provide  a  basis 
for  developing  additional  handling  options  and 


identifying  new  risks.  The  key  to  the  monitor¬ 
ing  process  is  to  establish  a  cost,  schedule,  and 
performance  management  indicator  system  over 
the  entire  program  that  the  PM  uses  to  evaluate 
the  status  of  the  program.  The  indicator  system 
should  be  designed  to  provide  early  warning  of 
potential  problems  to  allow  management  actions. 
Risk  monitoring  is  not  a  problem-solving  tech¬ 
nique,  but  rather,  a  proactive  technique  to  observe 
the  results  of  risk  handling  and  identify  new  risks. 
Some  monitoring  techniques  can  be  adapted  to 
become  part  of  a  risk  indicator  system: 

•  Test  and  Evaluation  (T &E) .  A  well-defined 
(T&E)  program  is  a  key  element  in  monitoring 
the  performance  of  selected  risk-handling  op¬ 
tions  and  developing  new  risk  assessments. 

•  Earned  Value  (EV) .  This  uses  standard  DoD 
cost/schedule  data  to  evaluate  a  program’s 
cost  and  schedule  performance  in  an  inte¬ 
grated  fashion.  As  such,  it  provides  a  basis 
to  determine  if  risk-handling  actions  are 
achieving  their  forecasted  results. 

•  Technical  Performance  Measurement 
(TPM).  TPM  is  a  product  design  assessment 
which  estimates,  through  engineering  analy¬ 
sis  and  tests,  the  values  of  essential  perfor¬ 
mance  parameters  of  the  current  design  as 
effected  by  risk-handling  actions. 

•  Program  Metrics.  These  are  used  for  for¬ 
mal,  periodic  performance  assessments  of 
the  various  development  processes,  evaluat¬ 
ing  how  well  the  system  development  pro¬ 
cess  is  achieving  its  objective.  This  technique 
can  be  used  to  monitor  corrective  actions  that 
emerged  from  an  assessment  of  the  critical 
risk  processes. 

•  Schedule  Performance  Monitoring.  This 
is  the  use  of  program  schedule  data  to  evalu¬ 
ate  how  well  the  program  is  progressing  to 
completion. 
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Paragraph  5.7  describes  several  monitoring  tech¬ 
niques,  e.g.,  earned  value,  etc. 

The  indicator  system  and  periodic  reassessments 
of  program  risk  should  provide  the  PMO  with 
the  means  to  incorporate  risk  management  into 
the  overall  program  management  structure. 

2.9  RISK  DOCUMENTATION 

A  primary  criteria  for  successful  management  is 
formally  documenting  the  ongoing  risk 
management  process.  This  is  important  because: 

•  It  provides  the  basis  for  program  assessments 
and  updates  as  the  program  progresses. 

•  Formal  documentation  tends  to  ensure  more 
comprehensive  risk  assessments  than  if  it  is 
not  documented. 

•  It  provides  a  basis  for  monitoring  risk¬ 
handling  actions  and  verifying  the  results. 

•  It  provides  program  background  material  for 
new  personnel. 

•  It  is  a  management  tool  for  the  execution  of 
the  program. 

•  It  provides  the  rationale  for  program 
decisions. 

The  documentation  should  be  done  by  those 
responsible  for  planning,  collecting,  and 
analyzing  data,  i.e.,  IPT  level  in  most  cases. 

Risk  management  reports  vary  depending  on 
the  size,  nature,  and  phase  of  the  program. 
Examples  of  some  risk  management  documents 
and  reports  that  may  be  useful  to  a  PM  are: 

•  Risk  management  plan, 

•  Risk  information  form, 


•  Risk  assessment  report, 

•  Prioritized  list  of  risks, 

•  Risk  handling  plan, 

•  Aggregated  risk  list, 

•  Risk  monitoring  documentation: 

-  Program  metrics, 

-  Technical  reports, 

-  Earned  value  reports, 

-  Watch  list, 

-  Schedule  performance  report, 

-  Critical  risk  processes  reports. 

Most  PMOs  can  devise  a  list  of  standard  reports 
that  will  satisfy  their  needs  most  of  the  time;  how¬ 
ever,  since  there  will  always  be  a  need  for  ad 
hoc  reports,  briefings,  and  assessments,  it  is  ad¬ 
visable  to  store  risk  information  in  a  management 
information  system  (MIS).  This  allows  the  cre¬ 
ation  of  both  standard  and  ad  hoc  reports,  as 
needed.  Paragraphs  4.8  and  5.8  discuss  an  MIS 
to  support  a  risk  management  program. 

Acquisition  reform  discourages  Government 
oversight;  therefore,  formal  contractor- pro¬ 
duced  risk  documentation  may  not  be  available 
for  most  programs.  However,  program  insight 
is  encouraged,  and  PMOs  can  obtain  infor¬ 
mation  about  program  risk  from  contractor 
internal  documentation  such  as: 

•  Risk  Management  Policy  and  Procedures . 

This  is  a  description  of  the  contractor’s  cor¬ 
porate  policy  for  the  management  of  risk.  The 
procedures  describe  the  methods  for  risk  iden¬ 
tification,  analysis,  handling,  monitoring,  and 
documentation.  It  should  provide  the  baseline 
planning  document  for  the  contractor’s 
approach  to  risk  management. 

•  Corporate  Policy  and  Procedures  Docu¬ 
ments.  Corporations  have  policy  and 


24 


rocedures  documents  that  address  the  func¬ 
tional  areas  that  are  critical  to  the  design,  en¬ 
gineering,  manufacture,  test  and  evaluation, 
quality,  configuration  control,  manufacture, 
etc.,  of  a  system.  These  documents  are  based 
on  what  the  company  perceives  as  best  prac¬ 
tices,  and  although  they  may  not  specifically 
address  risk,  deviation  from  these  policies  rep¬ 
resents  risk  to  a  program.  Internal  company 


reports  that  address  how  well  programs  com¬ 
ply  with  policy  may  be  required  and  will  pro¬ 
vide  valuable  information. 

•  Risk  Monitoring  Report.  Contractors 
should  have  internal  tracking  metrics  and 
reports  for  each  moderate-  or  high-risk  item. 
These  metrics  may  be  used  to  determine  the 
status  of  risk  reduction  programs. 
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3 

RISK  MANAGEMENT  AND  THE 
DOD  ACQUISITION  PROCESS 


3.1  INTRODUCTION 

This  Chapter  discusses  the  relationship  between 
risk  and  the  acquisition  process,  describes  how 
risk  is  considered  in  design  of  the  Acquisition 
Plan,  and  expresses  the  need  to  consider  risk  as 
early  in  the  program  as  possible.  Appendix  A 
is  a  summary  of  the  risk  management  require¬ 
ments  that  are  contained  in  DoDD  5000.1, 
DoDI  5000.2,  Interim  Defense  Acquisition 
Guidebook  (ID AG),  DoD  5000.4,  and  DoD 
5000.4-M. 

3.2  OVERVIEW 

The  DoD  acquisition  process  for  the  manage¬ 
ment  of  programs  consists  of  a  series  of  phases 
designed  to  reduce  risk,  ensure  affordability, 
and  provide  adequate  information  for  decision 
making.  Acquisition  officials  are  encouraged 
to  tailor  programs  to  eliminate  phases  or  activi¬ 
ties  that  result  in  little  payoff  in  fielding  time 
or  cost  savings.  To  effectively  tailor  a  program, 
one  needs  to  understand  the  risks  present  in  the 
program  and  to  develop  a  plan  for  managing 
these  risks.  DoD  policy  calls  for  the  continual 
assessment  of  program  risks,  beginning  with 
the  initial  phase  of  an  acquisition  program,  and 
the  development  of  management  approaches 
before  any  decision  is  made  to  enter  all 
subsequent  phases. 


The  application  of  risk  management  processes 
(planning,  assessment,  identification,  analysis, 
handling,  and  monitoring)  is  particularly  impor¬ 
tant  during  Concept  Refinement  (CR)  and  Tech¬ 
nology  Development  (TD)  Phases  of  any  pro¬ 
gram,  when  alternatives  are  evaluated,  program 
objectives  are  established,  and  the  acquisition 
strategy  is  developed.  All  of  these  activities  re¬ 
quire  acceptance  of  some  level  of  risk  and  de¬ 
velopment  of  plans  to  manage  the  risk. 

As  a  program  evolves  into  subsequent  phases, 
the  nature  of  the  risk  management  effort  will 
change.  New  assessments  will  be  built  on 
previous  ones.  Risk  areas  will  become  more 
specific  as  the  system  is  defined. 

Risk  management  should  also  be  an  integral 
part  of  any  Source  Selection  process,  from  re¬ 
quest  for  proposal  (RFP)  preparation,  through 
proposal  evaluation,  and  after  contract  award. 
Thi'oughout  the  program  life,  IPT s  will  play  a  key 
role  in  risk  management  activities. 

3.3  DOD  ACQUISITION  PROCESS 

The  phases  and  milestones  of  the  acquisition 
process  provide  a  streamlined  structure  that 
emphasizes  risk  management  and  affordability. 
The  phases  are  a  logical  means  of  progressively 
translating  broadly-stated  mission  needs  into 
well-defined  system- specific  requirements,  and 
ultimately  into  operationally  effective,  suitable, 
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and  survivable  systems.  It  is  important  to 
remember  that  the  term  “system”  includes 
hardware,  software,  and  the  human  element. 
Each  phase  is  designed,  among  other  things,  to 
manage  risks.  Milestones  are  points  in  time  that 
allow  decision  makers  to  evaluate  the  program 
status  and  determine  if  the  program  should  pro¬ 
ceed  to  the  next  phase.  The  Milestone  Decision 
Authority  (MDA)  and  PM  tailor  milestones  and 
phases  so  that  each  milestone  decision  point 
allows  assessment  of  program  status  and  the  op¬ 
portunity  to  review  plans  for  the  next  phase  and 
beyond.  The  MDA  should  explicitly  address 
program  risks  and  the  adequacy  of  risk  man¬ 
agement  planning  during  the  milestone  reviews 
and  establish  exit  criteria  for  progression  to  the 
next  phase. 

The  contract  schedule  normally  allows  time  for 
milestone  decisions  before  spending  begins  in 
subsequent  phases  and  should  also  permit 
demonstration  of  the  exit  criteria  in  time  to  sup¬ 
port  the  milestone  review.  There  are  exceptions 
to  this  —  driven  by  funding  availability  and 
option  award  dates.  However,  the  objective  is  to 
provide  proper  fiscal  control  without  delaying 
the  acquisition  decisions  or  contracts  while  ad¬ 
equately  considering  risk. 

The  acquisition  strategy  defines  the  business  and 
technical  management  approach  to  meet  objec¬ 
tives  within  program  constraints  with  a  primary 
goal  to  minimize  the  time  and  cost  of  satisfying 
a  valid  need,  consistent  with  common  sense  and 
sound  business  practices.  A  PM  prepares  a  pre¬ 
liminary  acquisition  strategy  —  called  a  Tech¬ 
nology  Development  Strategy  (TDS)  —  at  Mile¬ 
stone  A  (that  includes  TD  Phase  activities  that 
focus  on  identifying  technical  risk  and  handling 
options).  Later,  the  PM  updates  the  TDS  at  Mile¬ 
stone  B  into  a  Program  Acquisition  Strategy. 
This  strategy  is  updated  to  support  each  mile¬ 
stone  decision  by  describing  activities  and  events 
planned  for  the  upcoming  phase  and  relating  the 
accomplishments  of  that  phase  to  the  program’s 


overall,  long-term  objectives.  The  risk  associated 
with  a  program  will  significantly  influence  the 
acquisition  strategy. 

3.4  CHARACTERISTICS  OF 

THE  ACQUISITION  PROCESS 

The  acquisition  process  that  has  evolved  can 
be  characterized  in  terms  of  the  following 
concepts  that  are  particularly  relevant  to  the 
management  of  risk  in  programs. 

3.4.1  Integrated  Product  and  Process 
Development  (IPPD) 

IPPD  integrates  all  acquisition  activities  in  order 
to  optimize  system  development,  production, 
and  deployment.  Key  to  the  success  of  the  IPPD 
concept  are  the  IPTs,  which  are  composed  of 
qualified  and  empowered  representatives  from 
all  appropriate  functional  disciplines  who  work 
together  to  identify  and  resolve  issues.  As  such, 
IPTs  are  the  foundation  for  organizing  for  risk 
management. 

3.4.2  Continuous  Risk  Management 

PMs  should  focus  on  risk  management  through¬ 
out  the  life  of  the  program,  not  just  in  prepara¬ 
tion  for  program  and  milestone  reviews.  Pro¬ 
gram  risks  should  be  continuously  assessed,  and 
the  risk-handling  approaches  developed,  exe¬ 
cuted,  and  monitored  throughout  the  acquisi¬ 
tion  process.  Both  the  Government  and  contrac¬ 
tors  must  understand  risks  as  a  program 
progresses  through  the  various  phases  and  mile¬ 
stone  decision  points,  and  must  modify  the  man¬ 
agement  strategy  and  plan  accordingly.  While 
specific  government  and  contractors  risk  man¬ 
agement  processes  may  likely  be  different,  it  is 
important  that  each  party  have  a  common  and 
complete  set  of  process  steps  (regardless  of  their 
names),  and  be  able  to  exchange  and  clearly  un¬ 
derstand  the  other  party’s  risk  management 
documentation. 
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3.4.3  Program  Stability 

Once  a  program  is  initiated,  program  stability  is 
a  top  priority.  Keys  to  creating  program  stability 
are  realistic  investment  planning  and  affordability 
assessments.  They  must  reflect  an  accurate  and 
comprehensive  understanding  of  existing  or  ex¬ 
pected  program  risks.  A  risk  management  strat¬ 
egy  must  be  developed  early  in  the  process,  be¬ 
fore  actually  initiating  the  program  to  ensure  it  is 
a  stable  one,  recognizing  that  key  issues  affect¬ 
ing  program  stability  may  be  external. 

3.4.4  Reduction  of  Life-Cycle  Costs 

DoD  considers  the  reduction  of  total  cost  to  ac¬ 
quire  and  operate  systems  while  maintaining  a 
high  level  of  performance  for  the  user  to  be  of 
highest  priority.  This  is  reflected,  in  part, 
through  the  introduction  of  the  “Cost  As  an  In¬ 
dependent  Variable”  (CAIV)  concept.  CAIV 
entails  setting  aggressive,  realistic  cost  objec¬ 
tives  early  in  an  acquisition  program  and  then 
managing  all  aspects  of  the  program  to  achieve 
those  objectives,  while  still  meeting  the  user’s 
performance  and  schedule  needs.  Inherent  in 
the  CAIV  concept  is  the  realization  that  risks 
must  be  understood,  taken,  and  managed  in 
order  to  achieve  cost,  schedule,  and  perfor¬ 
mance  objectives.  An  understanding  of  risk  is 
essential  to  setting  realistic  cost  objectives.  The 
PM  and  user  representatives  should  identify  risk 
and  cost  driving  requirements  during  the  gen¬ 
eration  of  the  Capability  Development  Docu¬ 
ment  (CDD)  (formerly  the  Operational  Require¬ 
ment  Document  (ORD))  in  order  to  know  where 
tradeoffs  may  be  necessary. 

3.4.5  Event-Oriented  Management 

Event-oriented  management  requires  that  de¬ 
cision  makers  base  their  decisions  on  signifi¬ 
cant  events  in  the  acquisition  life  cycle,  rather 
than  on  arbitrary  calendar  dates.  This  manage¬ 
ment  process  emphasizes  effective  acquisition 


planning  and  embodies  sound  risk  management. 
Decisions  to  proceed  with  a  program  should  be 
based  on  demonstration  of  performance,  through 
test  and  evaluation,  and  on  verification  that  pro¬ 
gram  risks  are  well-understood  and  are  being 
managed  effectively.  Attainment  of  agreed-upon 
exit  criteria  is  an  indication  that  the  PMO  is  man¬ 
aging  risk  effectively. 

3.4.6  Modeling  and  Simulation 

Properly  used,  models  and  simulations  can 
reduce  time,  resources,  and  acquisition  risk 
and  may  increase  the  quality  of  the  systems 
being  developed.  Users  of  these  models  and 
simulations  must  have  a  good  understanding  of 
their  capabilities  and  limitations  and  their 
applicability  to  the  issues  being  addressed. 

From  a  risk  perspective,  modeling  and  simula¬ 
tion  may  be  used  to  develop  alternative  con¬ 
cepts  during  system  design;  predict  perfor¬ 
mance  in  support  of  trade-off  studies;  evaluate 
system  design  and  support  preliminary  design 
reviews  during  design  development;  predict 
system  performance  and  supplement  live  tests 
during  testing;  examine  the  military  value  of 
the  system;  determine  the  impact  of  design 
changes;  hone  requirements;  and  develop  life- 
cycle  support  requirements  and  assessments. 

However,  a  key  limitation  through  models  and 
simulations  is  that  the  results  are  only  as 
accurate  and  certain  as  the  quality  of  the  under¬ 
lying  relationships  and  input  data.  Blindly  be¬ 
lieving  and  using  the  output  from  models  and 
simulations  should  never  be  done. 

3.5  RISK  MANAGEMENT 
ACTIVITIES  DURING 
ACQUISITION  PHASES 

Risk  management  activities  should  be  applied 
continuously  throughout  all  acquisition  process 
phases  and  in  the  technology  opportunities  and 
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requirements  activities  that  feed  into  the  pro¬ 
cess.  However,  because  of  the  difference  in 
available  information,  the  level  of  application 
and  detail  will  vary  for  technology  opportu¬ 
nity  activities  and  for  each  phase.  For  techno¬ 
logical  opportunity  activities  in  the  Technol¬ 
ogy  Development  (TD)  Phase,  DoD  uses 
three  mechanisms  to  transition  concepts  and 
technology  to  user  and  acquisition  customers: 
Advanced  Technology  Demonstrations 
(ATDs),  Advanced  Concept  Technology 
Demonstrations  (ACTDs),  and  Experiments. 
When  assessing  the  risk  of  these  mechanisms, 
descriptors  called  Technology  Readiness  Lev¬ 
els  (TRLs)  are  used.  TRLs  provide  consistent, 
uniform  descriptions  of  technical  maturity  — 
across  different  types  of  technologies.  Appen¬ 
dix  6  of  the  IDAG  (also  see  Appendix  A,  page 
A- 12  of  this  Guide)  contains  guidance  on  use 
of  TRLs. 

In  the  TD  Phase,  management  focuses  on  as¬ 
sessing  the  risks  in  the  alternative  concepts  avail¬ 
able  to  satisfy  users  needs  and  on  planning  a  strat¬ 
egy  to  address  those  risks.  Lor  each  of  the  sub¬ 
sequent  phases,  all  four  risk  management 
activities  may  be  applied  with  increasing  focus 
on  risk  handling  and  monitoring. 

The  PM  identifies  objectives,  alternatives,  and 
constraints  at  the  beginning  of  each  phase  of  a 
program  and  then  evaluates  alternatives,  identi¬ 
fies  sources  of  project  risk,  and  selects  a  strategy 
for  resolving  the  risks.  The  PMO  updates  the 
acquisition  strategy,  risk  assessments,  and  other 
aspects  of  program  planning,  based  on  analy¬ 
ses,  for  the  phase  of  the  acquisition. 

Developers  should  become  involved  in  the  risk 
management  process  at  the  beginning,  when 
users  define  performance  requirements,  and  con¬ 
tinue  during  the  acquisition  process  until  the  sys¬ 
tem  is  delivered.  The  early  identification  and 
assessment  of  critical  risks  allow  PMs  to  formu¬ 
late  handling  approaches  and  to  streamline  the 


program  definition  and  the  RLP  around  critical 
product  and  process  risks. 

The  following  paragraphs  address  risk  man¬ 
agement  in  the  different  phases  in  more  detail. 

3.5.1  Concept  Refinement  (CR)  and 
Technology  Development  (TD) 
Phases 

The  Concept  Refinement  (CR)  Phase  normally 
consists  of  studies  and  the  Analysis  of  Alterna¬ 
tives  (AoA)  that  define  and  evaluate  the  feasi¬ 
bility  of  alternative  concepts  and  provide  the 
basis  for  the  assessment  of  these  alternatives  in 
terms  of  their  advantages,  disadvantages,  and 
risk  levels.  In  addition  to  providing  input  to  the 
AoA,  the  PM  develops  a  Technology  Devel¬ 
opment  Strategy  (TDS)  for  the  TD  Phase.  Later, 
in  the  TD  Phase,  technology  demonstrations 
are  conducted  and  decisions  on  technology 
readiness  are  made.  A  program  acquisition  strat¬ 
egy  is  developed  and  an  Acquisition  Program 
Baseline  (APB)  and  exit  criteria  are  established 
for  the  System  Integration  (SI)  part  of  the  Sys¬ 
tem  Development  and  Demonstration  (SDD) 
Phase. 

The  APB  documents  the  most  important  per¬ 
formance,  cost,  and  schedule  objectives  and 
thresholds  for  the  selected  concepts.  The 
parameters  selected  are  such  that  are-evaluation 
of  alternative  concepts  is  appropriate  if  thresh¬ 
olds  are  not  met.  Exit  criteria  are  events  or  ac¬ 
complishments  that  allow  managers  to  track 
progress  in  critical  technical,  cost,  or  schedule 
risk  areas.  They  must  be  demonstrated  to  show 
that  a  program  is  on  track. 

In  defining  alternative  concepts,  PMs  should 
pay  particular  attention  to  the  threat  and  the 
user’s  requirements,  which  are  normally  stated 
in  broad  terms  at  this  time.  Risks  can  be  intro¬ 
duced  if  the  requirements  are  not  stable,  or  if 
they  are  overly  restrictive  and  contain  specific 
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technical  solutions.  Requirements  can  also  be 
significant  cost  and  schedule  risk  drivers  if  they 
require  a  level  of  performance  that  is  difficult 
to  achieve  within  the  program  budget  and  time 
constraints.  Such  drivers  need  to  be  identified 
as  early  in  the  program  as  possible. 

The  acquisition  strategy  should  address  the 
known  risks  for  each  alternative  concept,  and 
the  plans  to  handle  them,  including  specific 
events  intended  to  control  the  risks.  Similarly, 
the  T&E  strategy  should  reflect  how  T&E,  with 
the  use  of  M&S,  will  be  used  to  assess  risk 
levels  and  identify  new  or  suspected  risk  areas. 

A  risk  management  strategy,  derived  in  concert 
with  the  acquisition  strategy,  should  be  devel¬ 
oped  during  this  phase  and  revised  and  updated 
continually  throughout  the  program.  This  strat¬ 
egy  should  include  risk  management  planning 
that  clearly  defines  roles,  responsibilities,  au¬ 
thority,  and  documentation  for  program  reviews, 
risk  assessments,  and  risk  monitoring. 

3.5.2  Subsequent  Phases 

During  subsequent  phases,  concepts,  techno¬ 
logical  approaches,  and/or  design  approaches 
(selected  at  the  previous  milestone  decisions) 
are  pursued  to  define  the  program  and  program 
risks.  Selected  alternative  concepts  continue  to 
be  analyzed,  and  the  acquisition  strategy,  and 
the  various  strategies  and  plans  derived  from 
it,  continue  to  be  refined. 

Risk  management  efforts  in  these  phases  focus 
on:  understanding  critical  technology,  manufac¬ 
turing,  and  support  risks,  along  with  cost,  sched¬ 
ule,  and  performance  risks;  and  demonstrating 
that  they  are  being  controlled  before  moving  to 
the  next  milestone.  Note  that  the  accuracy  of  cost, 
schedule,  performance  risk  assessments  should 
improve  with  each  succeeding  program  phase 
(e.g.,  more  info,  better  design  documentation, 
etc.).  Thus,  particular  attention  should  be  placed 


on  handling  and  monitoring  activities.  Planning 
and  assessment  should  continue  as  new  infor¬ 
mation  becomes  available  and  new  risk  events 
are  identified. 

During  these  phases,  the  risk  management  pro¬ 
gram  should  be  carried  out  in  an  integrated  Gov¬ 
ernment-contractor  framework  to  the  extent  pos¬ 
sible,  that  allows  the  Government  to  manage 
program  risks,  with  the  contractor  responsible 
to  the  PM  for  product  and  process  risks  and  for 
maintaining  design  accountability.  Both  the 
Government  and  contractors  need  to  understand 
the  risks  clearly,  and  jointly  plan  management 
efforts.  In  any  event,  risk  management  needs 
to  be  tailored  to  each  program  and  contract 
type. 

3.6  RISK  MANAGEMENT  AND 
MILESTONE  DECISIONS 

Before  a  milestone  review,  the  PM  should 
update  risk  assessments,  explicitly  addressing 
the  risks  in  the  critical  areas,  such  as  threat, 
requirements,  technology,  etc.,  and  identify 
areas  of  moderate  or  high  risk. 

Each  critical  technical  assessment  should  be 
supported  by  subsystems’  risk  assessments, 
which  should  be  supported  by  design  reviews, 
test  results,  and  specific  analyses. 

The  PM  should  present  planned  risk-handling 
actions  for  moderate-  or  high-risk  areas  at  the 
milestone  review  to  determine  their  adequacy  and 
to  ensure  the  efficient  allocation  of  resources. 

3.7  RISK  MANAGEMENT  AND  THE 
ACQUISITION  STRATEGY 

In  addition  to  providing  the  framework  for 
program  planning  and  execution,  the  acquisi¬ 
tion  strategy  serves  several  purposes  that  are 
important  to  risk  management: 
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•  Provides  a  master  schedule  for  research, 
development,  test,  production,  deployment, 
and  critical  events  in  the  acquisition  cycle. 

•  Gives  a  master  checklist  of  the  important  is¬ 
sues  and  alternatives  that  must  be  addressed. 

•  Assists  in  prioritizing  and  integrating  func¬ 
tional  requirements,  evaluating  alternatives, 
and  providing  a  coordinated  approach  to 
integrate  diverse  functional  issues,  leading  to 
the  accomplishment  of  program  objectives. 

•  Documents  the  assumptions  and  guidelines 
that  led  to  the  initiation  and  direction  of  the 
program. 

•  Provides  the  basis  for  the  development  and  ex¬ 
ecution  of  the  various  subordinate  functional 
strategies  and  plans. 

The  strategy  structure  should  ensure  a  sound 
program  through  the  management  of  cost,  sche¬ 
dule,  and  performance  risk.  A  good  acquisition 
strategy  acknowledges  and  identifies  program 
risks  and  forms  the  basis  for  implementing  a 
forward-looking,  rather  than  reactive,  effective 
risk  management  effort. 

Acquisition  strategy  should  describe  how  risk 
is  to  be  handled  and  identify  which  risks  are  to 
be  shared  with  the  contractor  and  which  are  to 
be  retained  by  Government.  The  key  concept 
here  is  that  the  Government  shares  the  risk  with 
the  contractor,  but  does  not  transfer  risk  to  the 
contractor.  The  PMO  always  has  a  responsibil¬ 
ity  to  the  system  user  to  develop  a  capable  sys¬ 
tem  and  can  never  absolve  itself  of  that  respon¬ 
sibility.  Therefore,  all  program  risks,  whether 
primarily  managed  by  the  PMO  or  by  the  con¬ 
tractor,  must  be  assessed  and  managed  by  the 
PMO. 

Once  the  program  office  has  determined  how 
much  of  each  risk  is  to  be  shared  with  the 


contractor,  it  should  assess  the  total  risk  assumed 
by  the  developing  contractor  (including  subcon¬ 
tractors).  The  Government  should  not  require 
contractors  to  accept  financial  risks  that  are 
inconsistent  with  their  ability  to  handle  them. 
Financial  risks  are  driven,  in  large  measure,  by 
the  underlying  technical  and  programmatic  risks 
inherent  in  a  program.  The  Government  contract¬ 
ing  officer  should,  therefore,  select  the  proper 
type  of  contract  based  on  an  appropriate  risk 
assessment,  to  ensure  a  clear  relationship 
between  the  selected  contract  type  and  program 
risk.  An  example  would  be  the  use  of  cost- 
reimbursable-type  contracts  for  development 
projects. 

3.8  RISK  MANAGEMENT  AND  CAIV 

The  intention  of  CAIV  is  to  establish  balance 
between  cost,  schedule,  performance,  and  risk 
early  in  the  acquisition  process  and  to  manage 
to  a  cost  objective.  CAIV  requires  that  PMs 
establish  aggressive  cost  objectives,  defined 
to  some  degree  by  the  maximum  level  of 
acceptable  risk.  Risks  in  achieving  both  per¬ 
formance  and  aggressive  cost  goals  must  be 
clearly  recognized  and  actively  managed 
through: 

(1)  continuing  iteration  of  cost/performance/ 
schedule/risk  tradeoffs, 

(2)  identifying  key  performance  and  manufac¬ 
turing  process  uncertainties,  and 

(3)  demonstrating  solutions  before  production. 

Whereas  DoD  has  traditionally  managed  per¬ 
formance  risk,  equal  emphasis  must  be  placed 
on  managing  cost  and  schedule  risks.  An  un¬ 
derlying  premise  of  CAIV  is  that  if  costs  are 
too  great,  and  there  are  ways  to  reduce  them, 
then  the  user  and  developer  may  reduce  perfor¬ 
mance  requirements  to  meet  cost  objections. 
Cost  control  and  effective  risk  management 


32 


involve  planning  and  scheduling  events  and  dem¬ 
onstrations  to  verify  solutions  to  cost,  schedule, 
and  performance  risk  issues. 

User  participation  in  the  trade-off  analysis  is 
essential  to  attain  a  favorable  balance  between 
cost,  schedule,  performance,  and  risk.  The  PM 
and  user  representatives  should  identify  risk 
and  cost  driving  requirements  during  the  gen¬ 
eration  of  the  CDD  to  know  where  tradeoffs 
may  be  possible.  Risk  assessments  are  critical 
to  the  CAIV  process  since  they  provide  users 
and  de-velopers  with  essential  data  to  assist 
in  the  cost,  schedule,  performance,  and  risk 
trade  decisions. 


Cost  for  risk  management  is  directly  related  to 
the  level  of  risk  and  affects  a  program  in  two 
ways.  First,  costs  are  associated  with  specific 
handling  activities,  for  example,  a  parallel 
development.  Second,  funds  are  needed  to 
cover  the  known  risks  of  the  selected  system 
approach  (i.e.,  funds  to  cover  cost  uncertainty). 
PMs  must  include  the  anticipated  expense  of 
managing  risk  in  their  estimates  of  program  costs. 
Decision  makers  must  weigh  these  costs  against 
the  level  of  risk  in  reaching  program  funding  de¬ 
cisions.  CAIV  requires  that  program  funds  sup¬ 
port  the  level  of  accepted  program  risk  and  that 
risk  management  costs  are  included  in  setting 
cost  objectives. 
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4 

RISK  MANAGEMENT 
AND 

PROGRAM  MANAGEMENT 


4.1  INTRODUCTION 

Risk  management  as  a  program  management 
responsibility  can  be  a  comprehensive  and 
responsive  management  tool  if  it  is  properly 
organized  and  monitored  at  the  PM  level.  A  for¬ 
malized  risk  management  program  should  be 
well-planned  and  forward-looking  by  identify¬ 
ing,  analyzing,  and  resolving  potential  problem 
areas  before  they  occur,  and  by  incorporating 
monitoring  techniques  that  accurately  portray 
the  status  of  risks  and  the  efforts  to  mitigate 
them.  Introduction  of  risk  management  early 
in  a  program  emphasizes  its  importance  and  en¬ 
courages  contractors  and  members  of  the 
Government  team  to  consider  risk  in  the  daily 
management  functions. 

This  Chapter  addresses  the  relationship  between 
risk  management  and  program  management  and 
suggests  methods  of  introducing  risk  manage¬ 
ment  in  a  program,  organizing  for  risk,  and 
training. 

4.2  OVERVIEW 

A  PMO  should  organize  for  risk  management, 
using  existing  IPTs.  The  PM  may  also  want  to 
use  contractors  to  support  management  efforts 
or  have  experts  not  involved  with  the  program 
perform  independent  assessments. 


To  use  risk  management  as  a  program  manage¬ 
ment  tool,  the  information  resulting  from  each 
of  the  risk  processes  should  be  documented  in 
a  usable  form  and  available  to  members  of  the 
Government/industry  program  team.  This  in¬ 
formation  will  provide  the  basis  for  reporting 
risk  and  overall  program  information,  both  in¬ 
ternally  and  externally.  Managing  collection 
and  dissemination  of  risk  information  can  be 
enhanced  through  the  use  of  a  Management 
Information  System  (MIS). 

4.3  PROGRAM  MANAGER  AND  RISK 
MANAGEMENT 

All  PMs  are  responsible  for  establishing  and 
executing  a  risk  management  program  that  sat¬ 
isfies  the  policies  contained  in  DoDD  5000.1 
and  DoDI  5000.2.  A  PM  must  balance  program- 
unique  requirements  or  circumstances  (e.g.,  size 
of  the  PMO  staff)  against  the  demands  of  proven 
risk  management  principles  and  practices.  This 
section  addresses  these  principles  and  practices 
and  provides  a  basis  for  establishing  a  PMO’s 
risk  management  organization  and  related  pro¬ 
cedures.  The  following  guidelines  define  an 
approach  to  risk  management. 


35 


4.3.1  Risk  Management  Is  a 
Program  Management  Tool 

Risk  management  should  be  integral  to  a 
program’s  overall  management.  PMs  must  take 
an  active  role  in  the  process  to  ensure  that  their 
approach  leads  to  a  balanced  use  of  program 
resources,  reflects  their  overall  management 
philosophy,  and  includes  Government  and  con¬ 
tractors.  Past  DoD  practices  have  generally 
treated  risk  management  solely  as  a  system 
engineering  function,  cost-estimating  technique 
or  possibly  as  an  independent  function  distinct 
from  other  program  functions.  Today,  risk  man¬ 
agement  is  recognized  as  a  vital  integrated  pro¬ 
gram  management  tool  that  cuts  across  the  en¬ 
tire  acquisition  program,  addressing  and  in¬ 
terrelating  cost,  schedule,  and  performance 
risks.  The  goal  is  to  make  everyone  involved  in 
a  program  aware  that  risk  should  be  a  consider¬ 
ation  in  the  design,  development,  and  fielding 
of  a  system.  It  should  not  be  treated  as  some¬ 
one  else’s  responsibility.  Specific  functional 
areas — such  as  system  engineering — could  be 
charged  with  implementing  risk  management, 
as  long  as  they  take  the  program  management 
view  towards  it. 

4.3.2  Risk  Management  Is  a 
Formal  Process 

Formal  risk  management  refers  to  a  structured 
process  whereby  risks  are  systematically  identi¬ 
fied,  analyzed,  handled,  and  monitored.  (A  rec¬ 
ommended  structure  is  described  in  Section  2  of 
this  Guide.)  A  stmctured  risk  management  pro¬ 
cess,  which  is  applied  early,  continuously,  and  rig¬ 
orously,  provides  a  disciplined  environment  for 
decision  making  and  for  the  efficient  use  of  pro¬ 
gram  resources.  Through  a  disciplined  process 
PMs  can  uncover  obscure  and  lower- level  risks 
that  collectively  could  pose  a  major  risk. 

The  need  for  a  formal  risk  management  process 
arises  from  the  nature  of  risk  and  the  complexity 


of  acquisition  programs.  The  numerous  risks  in 
an  acquisition  program  are  often  interrelated  and 
obscure  and  change  in  the  course  of  the  devel¬ 
opment  process.  A  formal  approach  is  the  only 
effective  method  to  sort  through  numerous  risk 
events,  to  identify  the  risks  and  their  interrela¬ 
tionships,  to  pinpoint  the  truly  critical  ones,  and 
to  identify  cost-effective  ways  to  reduce  those 
risks,  consistent  with  overall  program  objectives. 

A  structured  process  can  reduce  the  complexity 
of  an  acquisition  program  by  defining  an  ap¬ 
proach  to  assess,  handle,  monitor,  and  commu¬ 
nicate  program  risk.  The  systematic  identifica¬ 
tion,  analysis,  and  handling  of  risks  also  offers  a 
reliable  way  to  ensure  objectivity,  that  is,  mini¬ 
mize  unwarranted  optimism,  prejudice,  igno¬ 
rance,  or  self-interest.  Further,  structure  reduces 
the  impact  of  personnel  turnover  and  provides  a 
basis  for  training  and  consistency  among  all  the 
functional  areas  of  a  program.  A  structured  risk 
program  may  also  promote  teamwork  and  un¬ 
derstanding  and  improves  the  quality  of  the  risk 
products. 

4.3.3  Risk  Management  Is 
Forward-Looking 

Effective  risk  management  is  based  on  the 
premise  that  PMs  must  identify  potential  prob¬ 
lems,  referred  to  as  risk  events,  long  before  they 
can  occur  and  develop  strategies  that  increase 
the  probability/likelihood  of  a  favorable  outcome 
to  these  problems.  Application  of  this  philosophy 
occurs  primarily  by  using  analytical  techniques 
that  give  forward-looking  assessments. 

Typically,  the  early  identification  of  potential 
problems  is  concerned  with  two  types  of  events. 
The  first  are  relevant  to  the  current  or  immi¬ 
nent  acquisition  phase  of  a  program  (interme¬ 
diate-term),  such  as  satisfying  a  technical  exit 
criteria  in  time  for  the  next  milestone  review. 
The  second  are  concerned  with  the  future 
phase(s)  of  a  program  (long-term)  such  as 
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potential  risk  events  related  to  transitioning  a  sys¬ 
tem  from  development  to  production. 

By  analyzing  critical  events,  certain  risks  can 
be  determined.  To  do  this,  one  should  consider 
the  range  of  potential  outcomes  and  the  factors 
that  determine  those  outcomes.  Through  risk 
handling,  a  PM  then  develops  approaches  that 
minimize  risk  factors.  Paragraph  5.6  of  this 
Guide  describes  some  handling  approaches. 

Choosing  the  proper  risk-handling  options 
requires  that  a  balance  be  struck  between  the 
resources  required  to  implement  those  options 
and  their  payoffs  (both  intermediate  and  long¬ 
term)  and  the  resources  realistically  available. 

4.3.4  Risk  Management  Is  Integral  to 
Integrated  Product  and  Process 
Development  (IPPD) 

One  of  the  tenets  of  IPPD  is  multidisciplinary 
teamwork  through  IPTs,  which  are  an  integral 
part  of  the  defense  acquisition  oversight  and 
review  process.  The  Integrating  IPT  (IIPT)  is  a 
valuable  resource  to  assist  in  developing  a  risk 
management  plan  and  should  be  used  accord¬ 
ingly.  The  PM  should  ensure  that  the  require¬ 
ments  of  the  Overarching  IPT  (OIPT)  are 
reflected  in  the  plan. 

Working  with  the  OIPT,  the  PM  can  establish 
the  type  and  frequency  of  risk  management 
information  that  an  OIPT  requires,  and  refine 
management  organization  and  procedures. 
This  should  be  done  during  the  initial  OIPT 
meetings.  OIPTs  will  most  likely  require 
information  concerning: 

•  Known  risks  and  their  characteristics,  e.g., 
probability  of  occurrence  and  consequences/ 
impacts, 

•  Planned  risk-handling  actions,  funded  and 
unfunded, 


•  Achievements  in  controlling  risks  at  accept¬ 
able  levels. 

IIPTs  and  OIPTs  may  also  require  details  on 
the  PM’s  risk  management  program,  access  to 
the  risk  management  plan,  and  the  results  of 
specific  risk  assessments.  In  addition,  PMs  may 
want  to  present  selected  information  to  IIPTs 
and  OIPTs  to  help  substantiate  a  position  or 
recommendation,  e.g.,  help  support  a  budget 
request. 

4.4  RISK  MANAGEMENT 

ORGANIZATION  IN  THE  PMO 

The  PM,  after  determining  a  preferred  manage¬ 
ment  approach,  must  organize  the  program 
office  and  establish  outside  relationships  in 
order  to  manage  risk.  No  particular  organiza¬ 
tional  structure  is  superior;  however,  experience 
provides  some  insights  into  the  development  of 
effective  risk  management  organizations.  PMs 
should  consider  the  following  discussion  in  the 
context  of  their  unique  requirements  and 
circumstances  and  apply  those  that  are  suitable 
to  their  specific  needs. 

4.4.1  Risk  Management 

Organizational  Structure 

A  major  choice  for  each  PM  is  whether  to  have 
a  centralized  or  decentralized  risk  management 
organization.  The  PM  may  choose  a  centralized 
organizational  structure  until  team  members  be¬ 
come  familiar  with  both  the  program  and  the  risk 
management  process.  In  a  centralized  approach, 
the  PM  establishes  a  team  that  is  responsible  for 
all  aspects  of  risk  management.  The  team  would 
write  a  plan,  conduct  assessments,  evaluate  risk¬ 
handling  options,  and  monitor  progress.  Al¬ 
though  this  approach  may  be  necessary  early  in 
a  program,  it  tends  to  minimize  the  concept  that 
risk  management  is  a  responsibility  shared  by 
all  members  of  the  acquisition  team,  whether 
Government  or  contractor. 
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Figure  4-1.  Decentralized  Risk  Management  Organization 


The  PM  may  also  choose  to  decentralize.  The 
degree  of  decentralization  depends  on  the 
assignment  of  responsibilities.  Some  level  of 
centralization  is  almost  always  essential  for  prior¬ 
itizing  risk  across  the  program.  A  program  level 
IPT  (see  Figure  4-1)  or  a  Risk  Management 
Board  (RMB)  may  be  appropriate  for  this  inte¬ 
grating  function. 

The  decentralized  risk  management  organization 
is  the  most  widely  used  approach,  which  is  com¬ 
patible  with  the  DoD’s  IPPD  policy  and  gener¬ 
ally  results  in  an  efficient  use  of  personnel  re¬ 
sources.  In  this  approach,  risk  management  is 
delegated  to  Program  IPTs  (PIPTs). 

The  following  guidelines  apply  to  all  risk 
management  organizations: 

•  The  PM  is  ultimately  responsible  for  plan¬ 
ning,  allocating  resources,  and  executing  risk 
management.  This  requires  the  PM  to  over¬ 
see  and  participate  in  the  risk  management 
process. 


•  The  PM  must  make  optimal  use  of  available 
resources,  i.e.,  personnel,  organizations,  and 
funds.  Personnel  and  organizational  resources 
include  the  PMO,  functional  support  offices 
of  the  host  command,  the  prime  contractor, 
independent  risk  assessors,  and  support  con¬ 
tractors. 

•  Risk  management  is  a  team  function.  This 
stems  from  the  pervasive  nature  of  risk  and 
the  impact  that  risk-handling  plans  may  have 
on  other  program  plans  and  actions.  In  the 
aggregate,  risk  planning,  risk  assessment,  risk 
handling,  and  risk  monitoring  affect  all  pro¬ 
gram  activities  and  organizations.  Any  attempt 
to  implement  an  aggressive  forward-looking 
risk  management  program  without  the  in¬ 
volvement  of  all  PMO  subordinate 
organizations  could  result  in  confusion,  mis¬ 
direction,  and  wasted  resources.  The  only 
way  to  avoid  this  is  through  teamwork  among 
the  PMO  organizations  and  the  prime  con¬ 
tractor.  The  management  organizational  struc¬ 
ture  can  promote  teamwork  by  requiring 
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strong  connectivity  between  that  structure,  the 
various  PMO  organizations,  and  the  prime 
contractor.  The  teams  may  use  independent 
assessments  to  assist  them,  when  required. 

Figure  4- 1  portrays  a  decentralized  risk  manage¬ 
ment  organization.  This  example  includes  the  en¬ 
tire  PMO  and  selected  non-PMO  organizations, 
e.g.,  the  prime  contractor,  who  are  members  of 
the  IPTs.  The  figure  shows  that  risk  manage¬ 
ment  is  an  integral  part  of  program  management 
and  not  an  additional  or  separate  function  to  per¬ 
form.  Hence,  separate  personnel  are  not  desig¬ 
nated  to  manage  risk,  but  rather  all  individuals 
are  required  to  consider  risk  management  as  a 
routine  part  of  their  jobs.  In  the  figure,  the  risk 
coordinator  reports  to  the  PM,  but  works  in  co¬ 
ordination  with  the  PIPT,  functional  offices,  and 
the  Program  Level  IPT.  As  shown,  this  organi¬ 
zational  stmcture  is  suited  to  Acquisition  Cat¬ 
egory  (ACAT)  I  programs,  but  PMs  can  tailor  it 
to  satisfy  their  specific  requirements.  The  details 
are  dependant  upon  the  contract,  type,  statement 
of  work,  and  other  variables. 

The  organizational  structure  shows  that  the  PM 
is  ultimately  responsible  for  risk  management. 
There  is  a  coordinator  to  assist  with  this  respon¬ 
sibility  and  act  as  an  “operations”  officer.  This 
may  be  a  full-time  position  or  an  additional  duty 
as  the  PM  deems  appropriate.  The  coordinator 
should  have  specific  training  and  experience  in 
risk  management  to  increase  the  chance  of 
successful  implementation  and  to  avoid  common 
problems.  A  support  contractor  may  assist  the 
coordinator  by  performing  administrative  tasks 
associated  with  that  office. 

The  Program  Level  IPT,  composed  of  individu¬ 
als  from  the  PMO  and  prime  contractor,  ensures 
that  the  PM’s  risk  management  program  is  imple¬ 
mented  and  program  results  are  synthesized  into 
a  form  suitable  for  decision  making  by  the  PM 
and  OIPT. 


The  inclusion  of  both  Sub-Tier  IPTs  and  PMO 
functional  offices  simply  reflects  that  not  all  pro¬ 
gram  management  functions  will  be  assigned  to 
Sub-Tier  IPTs  for  execution. 

Independent  risk  assessors  are  typically  hired 
when  the  PM  has  specific  cost,  schedule,  per¬ 
formance  concerns  with  a  hardware  or  software 
product  or  engineering  process  and  wants  an 
independent  assessment  from  an  expert  in  a  par¬ 
ticular  field.  The  duration  of  their  services  is 
normally  short,  and  tailored  to  each  program. 

4.4.2  Risk  Management  Responsibilities 

This  section  identifies  the  primary  responsibili¬ 
ties  that  could  be  associated  with  a  decentral¬ 
ized  risk  management  organization.  In  assign¬ 
ing  the  responsibilities  to  the  various  organiza¬ 
tional  elements,  the  PM  should  strike  a  balance 
between  a  concentration  of  responsibilities  at 
the  higher  levels  and  pushing  them  too  far  down 
the  organizational  structure. 

The  development  of  these  responsibilities,  in  part, 
is  based  on  the  premise  that  risk  management 
activities  must  be  specific — and  assigned  to  in¬ 
dividuals,  not  groups.  The  responsibilities  listed 
below  are  assigned  to  the  leader  of  each  organi¬ 
zational  element,  recognizing  that  the  composi¬ 
tion  of  each  element  will  be  program  unique, 
i.e.,  number  of  assigned  PMO  personnel,  prime 
contractor  personnel,  etc.  The  task  of  further 
assigning  these  responsibilities,  along  with  tai¬ 
loring  them  to  satisfy  the  needs  and  requirements 
of  each  program,  remains  for  PMs  and  their 
staffs  to  accomplish. 

Table  4-1  provides  a  description  of  the  respon¬ 
sibilities  associated  with  the  decentralized  risk 
management  structure,  sorted  by  notional  orga¬ 
nizational  elements  that  may  make  up  the  risk 
management  structure. 
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Personnel 

Job  Responsibility 

Program 

Manager 

•  Plan,  organize,  direct,  and  control  risk  management. 

•  Comply  with  DoDD  5000.1 ,  DoDI  5000.2,  DoD  5000. 2-R,  DoDD  5000.4,  and  DoD 
5000.4-M  risk  management  guidance. 

•  Ensure  that  funds  are  available  to  support  approved  risk-handling  plans. 

•  Inform  and  advise  MDA,  Cost  Analysis  Improvement  Group  (CAIG)  and  OIPT  on 
program  risk  and  its  handling. 

Risk 

Management 

Coordinator 

•  Develop  and  maintain  risk  management  plans. 

•  Provide  risk  management  training. 

•  Define  the  risk  reporting  scales  to  be  used  by  the  program. 

•  Develop  and  maintain  a  risk  management  information  system. 

•  Prepare  risk  management  reports. 

•  Monitor  compliance  with  DoDD  risk  management  requirements. 

•  Ensure  that  risk  management  functions  and  tasks  performed  by  the  Sub-Tier  IPTs 
and  the  PMO  functional  offices  are  fully  integrated  and  in  compliance  with 
assigned  tasks. 

•  Advise  the  PM  and  Program  Level  IPT  on  the  use  of  risk  management  sources, 
i.e.,  host  command  functional  support  offices,  etc. 

•  Evaluate  risk  assessments,  risk-handling  plans,  and  risk  monitoring  results  as 
directed  and  recommend  appropriate  actions. 

•  Advise  the  PM  on  the  use  of  independent  risk  assessors. 

Program  Level 
IPT 

(some  PMOs 
use  a  Risk 
Management 
Board  (RMB) 
for  this 

responsibility) 

•  Ensure  that  the  risk  management  program  is  implemented,  risk  reduction  is 
accomplished  in  conformance  with  the  PM’s  strategy,  and  the  risk  management 
efforts  of  the  Sub-Tier  IPTs  are  integrated. 

•  Report  risk  events  to  the  risk  management  coordinator. 

•  Evaluate  whether  Sub-Tier  IPTs  and  PMO  functional  offices  have  identified 
critical  risks  and  proposed  risk-handling  plans. 

•  Ensure  that  cost,  schedule,  and  performance  risks  are  compatible. 

•  Ensure  that  cost,  schedule,  and  performance  risks  are  combined  in  a  manner 
consistent  with  the  plan. 

PMO  Sub-Tier 
PIPTs  & 
Functional 
Offices 

(Process) and 
System 
Elements 
(Products) 

•  Assess  risks,  recommending  appropriate  risk-handling  strategies  for  each 
identified  moderate  and  high  risk,  and  implementing  and  documenting  all  risk 
management  analyses  and  findings  within  the  team’s  product  area. 

•  Coordinate  all  risk  management  findings  and  decisions  with  other  Sub-Tier  IPTs, 
PMO  functional  offices,  the  Program  Level  IPT,  and  the  risk-management 
coordination  office. 

•  Identify  funding  requirements  to  implement  risk-handling  plans. 

•  Identify  the  need  for  risk  management  training. 

•  Report  risk  events  to  the  Program  Level  IPT  and  risk  coordinator. 

Independent 

Risk 

Assessors 

•  Perform  independent  risk  assessment  on  critical  risk  areas  or  contractor 
engineering  processes  that  the  PM  has  specified. 

•  Report  the  results  of  those  assessments  to  the  PM. 

•  Work  with  the  risk  management  coordinator. 

Table  4-1.  Notional  Description  of  Risk  Management  Responsibilities 
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4.5  CONTRACTOR  RISK 
MANAGEMENT 

Experience  has  shown  that  managing  a  program’s 
risks  requires  a  close  partnership  between  the 
PMO  and  the  prime  contractor(s).  PMs  must  de¬ 
termine  the  type  of  support  they  need  from  their 
prime  contractor,  communicate  these  needs 
through  the  Request  for  Proposal  (RFP)  for  each 
acquisition  phase,  and  then  provide  for  them  in 
the  contract.  Preparation  of  the  RFP  and  source 
selection  are  discussed  in  subsequent  sections. 

4.5.1  Contractor  View  of  Risk 

Contractors  treat  risk  differently  from  the  Gov¬ 
ernment  because  each  views  risk  from  a  differ¬ 
ent  perspective.  The  PM,  in  executing  his  risk 
management  program,  needs  to  understand  the 
contractor  viewpoint. 

Contractors  typically  divide  risks  into  two  basic 
types:  business  risks  and  program  risks.  Busi¬ 
ness  risk,  in  the  broadest  sense,  involves  the 
inherent  chance  of  making  a  profit  or  incurring 
a  loss  on  any  given  contract.  Program  risk  in¬ 
volves,  among  other  things,  technical,  require¬ 
ment,  and  design  uncertainties.  A  contractor’s 
efforts  to  minimize  business  risks  may  conflict 
with  a  Government  PM’s  efforts  to  lower 
program  risk. 

While  the  government  and  contractors  may  have 
different  views  on  specific  cost,  schedule,  and 
performance  risk  levels/ratings,  they  generally 
have  (or  should  have)  similar  views  of  the  risk 
management  process.  One  exception  may  be 
the  requirements  placed  by  corporate  manage¬ 
ment — that  could  conflict  with  the  Government 
view  of  program  risk.  The  similarity,  however, 
does  not  necessarily  lead  to  the  contractor  hav¬ 
ing  a  competent  internal  risk  management  pro¬ 
gram.  As  a  Project  Management  Institute  (PMI) 
handbook  points  out,  “On  most  (contractor)  pro¬ 
jects,  responsibility  for  Project  Risk  is  so 


pervasive  that  it  is  rarely  given  sufficient  central 
attention.”  As  a  minimum,  it  is  important  that  the 
PMO  writes  the  RFP  asking  the  contractor  to 
describe  its  risk  management  process,  including  its 
approach  to  managing  any  specific  areas. 

4.5.2  Govemment/Contractor 
Relationship 

The  prime  contractor’s  support  and  assistance 
is  required  even  though  the  ultimate  responsi¬ 
bility  for  risk  management  rests  with  the  Gov¬ 
ernment  PM.  Often,  the  contractor  is  better 
equipped  to  understand  the  program  technical 
risks  than  the  Government  program  office  is. 
Both  the  Government  and  contractor  need  to 
share  information,  understand  the  risks,  and 
develop  and  execute  management  efforts.  The 
Government  must  involve  the  contractor  early 
in  program  development,  so  that  effective  risk 
assessment  and  reduction  can  occur. 

Therefore,  risk  management  must  be  a  key  part 
of  the  contractor’s  management  scheme.  Al¬ 
though  the  Government  does  not  dictate  how 
the  contractor  should  manage  risk,  some  char¬ 
acteristics  of  a  good  Government/contractor 
relationship  include: 

•  Clear  definition  of  risks  and  their  assignment. 

•  Flexibility  for  assignment  of  risks  and  risk 
management  responsibilities  among  the 
teams. 

•  Strong  emphasis  on  best  management  and 
technical  practices  which,  if  followed,  avoid 
unnecessary  risks. 

Regarding  RFP  development,  discussed  later  in 
this  section,  information  is  provided  on  how 
these  characteristics  should  be  addressed. 

The  Govemment/contractor  partnership  can  be 
forged  in  at  least  two  ways.  First,  the  PMO 
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should  include  the  prime  contractor(s)  in  the  top- 
level  risk  planning  and  assessment  activities.  This 
includes  understanding  and  factoring  in  such  is¬ 
sues  as  user  requirements,  affordability  con¬ 
straints,  and  schedule  limitations.  Second,  the 
PMO  should  include  in  advance  specific  risk 
assessment  and  handling  tasks  as  key  contrac¬ 
tual  efforts  during  the  concept  exploration  and 
program  definition  and  risk  reduction  phases. 

Forming  a  joint  Government/contractor  evalu¬ 
ation  team  is  a  good  way  of  fostering  an  effec¬ 
tive  partnership.  This  is  especially  true  in  a 
program’s  early  stages  when  uncertainty  is  high 
and  both  parties  must  frequently  assess  risks. 
These  assessments,  properly  handled,  involve 
multidisciplinary  efforts  requiring  subject-mat¬ 
ter  experts  from  both  the  prime  contractor  and 
Government.  This  joint  team  should  evaluate 
the  proposed  program  in  detail  and  explore  the 
inherent  program  risks,  the  proposed  handling 
strategies,  the  detailed  development  schedule, 
and  the  contractor’s  developmental  resources 
(people,  facilities,  processes,  tools,  etc.). 

A  management  approach  using  multiple  teams 
is  the  best  approach  to  use,  e.g.,  Sub-Tier  IPTs. 
Joint  team(s)  should  be  established  at  the  be¬ 
ginning  of  each  development  phase  to  assess 
the  risks  to  be  overcome  in  that  phase  and  to 
determine  the  handling  technique(s)  to  be  used. 
Requirements  for  contractor  participation  on  the 
team(s)  should  be  identified  in  the  RFP  and  sub¬ 
sequent  contract. 

4.6  RISK  MANAGEMENT  AND  THE 
CONTRACTUAL  PROCESS 

4.6.1  Risk  Management: 

Pre-Contract  Award 

The  contractor’s  developmental  and  manufac¬ 
turing  processes  and  tools,  the  availability  and 
skill  of  personnel,  and  the  previous  experience 
of  the  Government  and  contractor  team  all  in¬ 


fluence  their  ability  to  handle  the  proposed  sys¬ 
tem  development  and  production.  Therefore,  an 
effective  risk  management  process  includes  an 
evaluation  of  the  capabilities  of  the  potential 
contractors. 

4.6.2  Early  Industry  Involvement: 
Industrial  Capabilities  Review 

An  Industrial  Capabilities  Review  is  a  powerful 
tool  available  to  PMs  for  determining  general 
industrial  capabilities.  To  avoid  potential  prob¬ 
lems  in  the  subsequent  competitive  process  and 
to  ensure  that  a  “level  playing  field”  is  main¬ 
tained,  an  announcement  in  the  Federal  Busi¬ 
ness  Opportunities  (FedBizOpps)  should  be 
made  to  inform  all  potential  offerors  that  the  Gov¬ 
ernment  plans  to  conduct  an  Industrial  Capabili¬ 
ties  Review  and  to  request  responses  from  all 
interested  parties.  Below  is  a  general  approach 
that  PMOs  may  find  readily  adaptable  to  any 
type  of  capability  review.  The  basic  steps  in  the 
process  are  to: 

•  Obtain  the  Source  Selection  Authority’s 
approval  to  conduct  the  review. 

•  Establish  the  criteria  for  the  capability. 

•  Identify  the  potential  contractors  who  will 
participate  in  the  review. 

•  Provide  an  advance  copy  of  the  review 
material  to  those  contractors. 

•  Select  the  review  team,  ensuring  that  it  has 
the  necessary  mix  of  talent. 

•  Train  the  team  on  the  purpose  of  the  review 
and  review  criteria. 

•  Conduct  the  review  and  evaluate  the  results. 

•  Provide  feedback  to  each  contractor  on  the 
results  of  their  review  and  assessment. 
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•  Provide  the  results  to  the  PM. 

This  review  is  an  appraisal  of  general  industrial 
capabilities  and  supports  identifying  potential 
program  risks  and  best  practices  rather  than 
evaluating  specific  contractors. 

Regardless  of  the  approach,  the  PMO  should 
determine  what  specific  information  is  needed. 
DoD  4245. 7-M  is  a  good  guide  to  help  tailor 
a  set  of  questions  for  the  contractors.  The 
questions  generally  focus  on  two  areas  consis¬ 
tent  with  protection  of  contractor  proprietary 
information. 

•  What  is  the  state-of-the-art  of  the  technology 
proposed  for  use  in  the  system? 

•  What  are  the  general  developmental/manu¬ 
facturing  capabilities  of  the  potential  con¬ 
tractors  (including  experience,  tools,  pro¬ 
cesses,  etc.)  as  compared  to  industry  best 
practices? 

Table  4-2  shows  some  of  the  specific  areas  or 
sources  for  risk  identification.  It  includes  a  num¬ 
ber  of  areas  (threat,  requirements,  design,  etc.)  that 
have  been  shown  through  experience  to  contain 
risk  events  that  tend  to  be  more  critical  than  oth¬ 
ers,  and  which  ones  should  receive  the  most  man¬ 
agement  attention.  Risk  events  are  determined  by 
examining  WBS  element  product  and  processes 
in  terms  of  risk  areas.  Process  areas  are  specifi¬ 
cally  addressed  in  DoD  4245. 7M.  They  are  gen¬ 
eral  in  that  areas  of  risk  could  be  present  in  any 
program  from  either  source  (WBS  or  process). 
They  are  intended  as  a  list  of  “top-level”  risk 
sources  that  will  focus  attention  on  a  specific  area. 
The  PMO  and  contractor(s)  will  have  to  examine 
lower  levels  to  understand  the  actual  risks  that  are 
present  in  their  program  and  to  develop  an  effec¬ 
tive  management  plan.  The  risks  shown  are  not 
intended  to  serve  as  a  simple  checklist  that  one 
should  apply  directly,  then  consider  the  program 
risk-free  if  none  of  the  listed  risks  are  present. 


An  examination  of  the  program  in  these  areas 
can  help  to  develop  the  final  program  acquisi¬ 
tion  strategy  and  the  risk- sharing  structure  be¬ 
tween  the  Government  and  industry.  The  PMO 
can  also  use  the  results  to  adjust  the  RFP  for 
the  next  phase  of  the  program. 

4.6.3  Developing  the 

Request  for  Proposal 

The  RFP  should  communicate  to  all  offerors 
the  concept  that  risk  management  is  an  essential 
part  of  the  Government’s  acquisition  strategy. 

Before  the  draft  RFP  is  developed  using  the  re¬ 
sults  of  the  Industrial  Capabilities  Review,  the 
PMO  should  conduct  a  risk  assessment  to 
ensure  that  the  program  described  in  the  RFP 
is  executable  within  the  technical,  schedule,  and 
budget  constraints.  Based  on  this  assessment,  a 
program  plan,  an  integrated  master  schedule, 
and  life-cycle  cost  (LCC)  estimate  may  be  pre¬ 
pared.  The  technical,  schedule,  and  cost  issues 
should  be  discussed  in  the  pre-proposal  con¬ 
ference^)  before  the  draft  RFP  is  released.  In 
this  way,  critical  risks  inherent  in  the  program 
can  be  identified  and  addressed  in  the  RFP.  In 
addition,  this  helps  to  establish  key  risk-man¬ 
agement  contractual  conditions.  The  RFP  should 
encourage  offerors  to  extend  the  contract  WBS 
(CWBS)  to  reflect  how  they  will  identify  all  el¬ 
ements  at  any  level  that  are  expected  to  be  high 
cost  or  high  risk.  The  RFP  should  also  encour¬ 
age  offerors  to  cite  any  elements  of  the  CWBS 
provided  in  the  draft  RFP  that  are  not  consistent 
with  their  planned  approach. 

In  the  solicitation,  PMs  may  ask  offerors  to  in¬ 
clude  a  risk  analysis  and  a  description  of  their 
management  plans,  and  also  to  develop  a  sup¬ 
porting  program  plan  and  an  integrated  master 
schedule  in  their  proposals.  These  proposals 
will  support  the  Government’s  source  selection 
evaluation  and  the  formulation  of  a  most  prob¬ 
able  cost  estimate  for  each  proposal.  In  addition, 
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Risk  Area 

Significant  Risks 

Threat 

•  Uncertainty  in  threat  accuracy. 

•  Sensitivity  of  design  and  technology  to  threat. 

•  Vulnerability  of  system  to  threat  and  threat  countermeasures. 

•  Vulnerability  of  program  to  intelligence  penetration. 

Requirements 

•  Operational  requirements  not  properly  established  or  vaguely  stated. 

•  Requirements  are  not  stable. 

•  Required  operating  environment  not  described. 

•  Requirements  do  not  address  logistics  and  suitability. 

•  Requirements  are  too  constrictive — identify  specific  solutions  that  force  high  cost. 

Design 

•  Design  implications  not  sufficiently  considered  in  concept  exploration. 

•  System  will  not  satisfy  user  requirements. 

•  Mismatch  of  user  manpower  or  skill  profiles  with  system  design  solution  or  human- 
machine  interface  problems. 

•  Increased  skills  or  more  training  requirements  identified  late  in  the  acquisition 
process. 

•  Design  not  cost  effective. 

•  Design  relies  on  immature  technologies  or  “exotic”  materials  to  achieve 
performance  objectives. 

•  Software  design,  coding,  and  testing. 

Test  and 
Evaluation 

•  Test  planning  not  initiated  early  in  program  (CR  Phase). 

•  Testing  does  not  address  the  ultimate  operating  environment. 

•  Test  procedures  do  not  address  all  major  performance  and  suitability 
specifications. 

•  Test  facilities  not  available  to  accomplish  specific  tests,  especially  system-level 
tests. 

•  Insufficient  time  to  test  thoroughly. 

Simulation 

•  Same  risks  as  contained  in  the  Significant  Risks  for  Test  and  Evaluation. 

•  M&S  are  not  verified,  validated,  or  accredited  for  the  intended  purpose. 

•  Program  lacks  proper  tools  and  modeling  and  simulation  capability  to  assess 
alternatives. 

Technology 

•  Program  depends  on  unproved  technology  for  success — there  are  no 
alternatives. 

•  Program  success  depends  on  achieving  advances  in  state-of-the-art  technology. 

•  Potential  advances  in  technology  will  result  in  less  than  optimal  cost-effective 
system  or  make  system  components  obsolete. 

•  Technology  has  not  been  demonstrated  in  required  operating  environment. 

•  Technology  relies  on  complex  hardware,  software,  or  integration  design. 

Logistics 

•  Inadequate  supportability  late  in  development  or  after  fielding,  resulting  in  need 
for  engineering  changes,  increased  costs,  and/or  schedule  delays. 

•  Life-cycle  costs  not  accurate  because  of  poor  logistics  supportability  analyses. 

•  Logistics  analyses  results  not  included  in  cost-performance  tradeoffs. 

•  Design  trade  studies  do  not  include  supportability  considerations. 

Table  4-2.  Significant  Risks  by  Critical  Risk  Areas 
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Risk  Area 

Significant  Risks 

Production/ 

Facilities 

•  Production  implications  not  considered  during  concept  exploration. 

•  Production  not  sufficiently  considered  during  design. 

•  Inadequate  planning  for  long  lead  items  and  vendor  support. 

•  Production  processes  not  proven. 

•  Prime  contractors  do  not  have  adequate  plans  for  managing  subcontractors. 

•  Sufficient  facilities  not  readily  available  for  cost-effective  production. 

•  Contract  offers  no  incentive  to  modernize  facilities  or  reduce  cost. 

Concurrency 

•  Immature  or  unproven  technologies  will  not  be  adequately  developed  before 
production. 

•  Production  funding  will  be  available  too  early — before  development  effort  has 
sufficiently  matured. 

•  Concurrency  established  without  clear  understanding  of  risks. 

Capability  of 
Developer 

•  Developer  has  limited  experience  in  specific  type  of  development. 

•  Contractor  has  poor  track  record  relative  to  costs  and  schedule. 

•  Contractor  experiences  loss  of  key  personnel. 

•  Prime  contractor  relies  excessively  on  subcontractors  for  major  development 
efforts. 

•  Contractor  will  require  significant  capitalization  to  meet  program  requirements. 

Cost/Funding 

•  Realistic  cost  objectives  not  established  early. 

•  Marginal  performance  capabilities  incorporated  at  excessive  costs;  satisfactory 
cost-performance  tradeoffs  not  done. 

•  Excessive  life-cycle  costs  due  to  inadequate  treatment  of  support  requirements. 

•  Significant  reliance  on  software. 

•  Funding  profile  does  not  match  acquisition  strategy. 

•  Funding  profile  not  stable  from  budget  cycle  to  budget  cycle. 

Schedule 

•  Schedule  not  considered  in  trade-off  studies. 

•  Schedule  does  not  reflect  realistic  acquisition  planning. 

•  APB  schedule  objectives  not  realistic  and  attainable. 

•  Resources  not  available  to  meet  schedule. 

Management 

•  Acquisition  strategy  does  not  give  adequate  consideration  to  various  essential 
elements,  e.g.,  mission  need,  test  and  evaluation,  technology,  etc. 

•  Subordinate  strategies  and  plans  are  not  developed  in  a  timely  manner  or  based 
on  the  acquisition  strategy. 

•  Proper  mix  (experience,  skills,  stability)  of  people  not  assigned  to  PMO  or  to 
contractor  team. 

•  Effective  risk  assessments  not  performed  or  results  not  understood  and  acted 
upon. 

Table  4-2.  Significant  Risks  by  Critical  Risk  Areas 
(continued) 

45 


the  RFP  may  identify  the  requirement  for  peri¬ 
odic  risk  assessment  reports  that  would  serve  as 
inputs  to  the  PM’s  assessment  and  monitoring 
processes  thereby  ensuring  that  risks  are  con¬ 
tinuously  assessed. 

4.6.4  The  Offeror’s  Proposal 

The  offerors  should  develop  the  proposed  pro¬ 
gram  plans  and  documentation  at  a  level  that  is 
adequate  to  identify  risks,  develop  associated 
management  activities  that  they  will  use  through¬ 
out  the  program,  and  integrate  resources,  tech¬ 
nical  performance  measures,  and  schedule  in 
the  proposed  program  plans.  Program  plans 
should  extend  the  CWBS  to  reflect  the  offeror’s 
approach  and  include  the  supporting  activities, 
critical  tasks,  and  processes  in  the  CWBS  dic¬ 
tionary.  The  associated  schedules  for  each 
should  be  incorporated  into  an  integrated  mas¬ 
ter  schedule.  Plans  should  also  have  an  estimate 
of  the  funds  required  to  execute  the  program 
and  include  a  breakout  of  resource  requirements 
for  high-risk  areas. 

The  information  required  and  the  level  of  de¬ 
tail  will  depend  on  the  acquisition  phase,  the 
category,  and  criticality  of  the  program,  as  well 
as  on  the  contract  type  and  value.  However,  the 
detail  submitted  with  the  proposal  must  be  at  a 
sufficiently  low  level  to  allow  identification  of 
possible  conflicts  in  the  planned  acquisition 
approach  and  to  support  the  Government’s  pro¬ 
posal  evaluation.  Generally,  the  CWBS  should 
be  defined  below  level  3,  by  the  contractor,  only 
to  the  extent  necessary  to  capture  those  lower 
level  elements  that  are  high  cost,  high  risk,  or 
of  high  management  interest. 

4.6.5  Basis  for  Selection 

DoD  acquisition  management  must  focus  on 
balancing  cost,  schedule,  performance,  and  risk 
by  selecting  the  contractor  team  that  provides 
the  best  value  to  the  user  within  acceptable  risk 


limits.  Therefore,  the  RFP/Source  Selection  pro¬ 
cess  must  evaluate  each  offeror’s  capability  for 
meeting  product  and  process  technical,  cost  and 
schedule  requirements  while  addressing  and  con¬ 
trolling  the  risks  inherent  in  a  program. 

The  evaluation  team  should  discriminate  among 
offerors  based  upon  the  following: 

•  Risks  determined  by  comparison  with  the 
best  practices  baseline. 

•  Ability  to  perform  with  a  focus  on  the  critical 
risk  elements  inherent  in  the  program. 

•  Adherence  to  requirements  associated  with 
any  mandatory  legal  items. 

•  Past  performance  on  efforts  similar  to  the 
proposed  program  being  evaluated. 

The  process  of  choosing  among  offerors  may 
be  enhanced  if  the  evaluation  team  includes  risk 
management  as  a  “source  selection  discrimi¬ 
nator.”  Risk  management  then  becomes  an 
important  factor  in  the  Source  Selection  Author¬ 
ity  determination  of  who  provides  the  most 
executable  program. 

4.6.6  Source  Selection 

The  purpose  of  a  source  selection  is  to  select  the 
contractor  whose  cost,  schedule  and  perfor¬ 
mance  can  best  be  expected  to  meet  the 
Government’s  requirements  at  an  affordable 
price.  To  perform  this  evaluation,  the  Govern¬ 
ment  must  assess  both  proposal  risk  and  per¬ 
formance  risk  for  each  proposal.  These  risk 
assessments  must  be  done  entirely  within  the 
boundaries  of  the  source  selection  process. 
Previous  assessments  of  any  of  the  offerors  may 
not  be  applicable  or  allowable. 

4.6.6. 1  Proposal  Risk.  This  refers  to  the  risk 
associated  with  the  offeror’s  proposed  approach 
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to  meet  the  Government  cost,  schedule,  and  per¬ 
formance  requirements.  The  evaluation  of  pro¬ 
posal  risk  includes  an  assessment  of  proposed 
time  and  resources  and  recommended  adjust¬ 
ments.  This  assessment  should  be  performed  ac¬ 
cording  to  the  definitions  and  evaluation  stan¬ 
dards  developed  for  the  source  selection.  Pro¬ 
posal  risk  is,  in  essence,  a  moderate  expansion 
of  past  evaluation  processes.  Historically,  evalu¬ 
ators  selected  contractors  who  demonstrated 
that  they  understood  the  requirements  and 
offered  the  best  value  approach  to  meeting 
the  Government’s  needs.  The  expansion  on  this 
concept  is  the  specific  consideration  of  risk. 

Technical  and  schedule  assessments  are  primary 
inputs  to  the  most  probable  cost  estimate  for 
each  proposal.  It  is  important  to  estimate  the 
additional  resources  needed  to  control  any  risks 
that  have  moderate  or  high  risk  ratings.  Offerors 
may  define  them  in  terms  of  additional  time, 
personnel  loading,  hardware,  or  special  actions 
such  as  additional  tests.  However,  whatever  the 
type  of  the  required  resources,  it  is  essential  that 
cost  estimates  be  integrated  and  consistent  with 
the  technical  and  schedule  evaluations. 

4.6.6.2  Performance  Risk.  A  performance  risk 
assessment  is  an  evaluation  of  the  contractor’s 
past  and  present  performance  record  to  establish 
a  level  of  confidence  in  the  contractor’s  ability 
to  perform  the  proposed  effort.  Such  an  evalua¬ 
tion  is  not  limited  to  programmatic  technical  is¬ 
sues,  but  also  includes  assessment  of  critical  ven¬ 
dor  financial  viability.  Financial  cap-ability  analy¬ 
ses  and  industrial  capability  assessments,  con¬ 
ducted  in  accordance  with  DoD  Handbook 
5000.60H,  provide  insight  to  a  contractor’s  ability 
to  perform  the  proposed  effort. 

A  range  of  methods  are  available  to  the  PM  to 
evaluate  performance  risk.  The  Performance 
Risk  Assessment  Group  (PRAG)  is  a  group  of 
experienced  Government  personnel  that  are  ap¬ 
pointed  by  the  source  selection  advisory  council 


Chairperson  to  permit  performance  risk  to  be 
used,  if  appropriate.  Performance  risk  may  be 
separately  assessed  for  each  evaluation  factor  or 
as  a  whole  with  the  assessment  provided  directly 
to  the  source  selection  advisory  council/author¬ 
ity  for  final  decision  or  indirectly  through  the 
Source  Selection  Evaluation  Board.  The 
assessment  relies  heavily  (although  not 
exclusively)  on  the  contractor  performance 
evaluations  and  surveys  submitted  by  the  PMO 
and  Defense  Contract  Management  Agency 
(DCMA). 

4.7  RISK  MANAGEMENT: 

POST-CONTRACT  AWARD 

Post-contract  award  risk  management  builds  on 
the  work  done  during  the  pre-contract  award 
phase.  With  the  award  of  the  contract,  the  rela¬ 
tionship  between  the  Government  and  the  con¬ 
tractor  changes  as  teams  are  formed  to  address 
program  risk.  These  teams  should  validate  pre¬ 
contract  award  management  plans  by  review¬ 
ing  assessments,  handling  plans,  and  monitor¬ 
ing  intentions.  The  extent  of  assessments  in¬ 
creases  as  the  contractor  develops  and  refines 
his  design,  test  and  evaluation,  and  manufac¬ 
turing  plans.  The  Government  PMO  should 
work  with  the  contractor  to  refine  handling 
plans. 

The  process  begins  with  an  Integrated  Baseline 
Review  (IBR)  after  contract  award  to  ensure  that 
reliable  plans  and  performance  measurement 
baselines  capture  the  entire  scope  of  work,  are  con¬ 
sistent  with  contract  schedule  requirements,  and 
have  adequate  resources  assigned  to  complete 
program  tasks.  The  IBR  could  be  conducted  to 
incorporate  other  steps  identified  below.  These 
steps  suggest  an  approach  that  the  PMO  might 
take  to  initiate  the  program’s  risk  management 
plans  and  activities  after  contract  award.  They 
are  intended  to  be  a  starting  point,  and  the  PMO 
should  tailor  the  plan  to  reflect  each  program’s 
unique  needs. 
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•  Conduct  initial  meeting  with  the  contractor 
to  describe  the  program’s  objectives  and 
approach  to  managing  risks.  The  PM  may  also 
present  the  risk  management  plan. 

•  Train  members  of  the  PMO  and  the  con¬ 
tractor’s  organization  on  risk  management 
basics,  incorporating  the  program’s  manage¬ 
ment  plan  and  procedures  into  the  training. 

•  Review  the  pre-contract  award  risk  plan  with 
the  PMO  and  contractor,  revise  it  as  neces¬ 
sary,  and  share  results  with  the  contractor. 

•  Conduct  in-depth  review  of  the  pre-contract 
award  risk  assessments  and  expand  the  review 
to  include  any  new  information  obtained  since 
the  award  of  the  contract. 

•  Review  and  revise  risk-handling  plans  to 
reflect  the  reassessment  of  risks. 

•  Review  the  program’s  documentation  require¬ 
ments  with  the  contractor.  Ensure  that  the 
PMO  and  contractor  understand  the  purpose, 
format,  and  contents  of  various  risk  reports. 

•  Initially,  it  may  be  necessary  to  establish  a 
formalized  PMO-contractor  risk  management 
organization  for  the  program,  consistent  with 
the  terms  of  the  contract. 

•  Working  with  the  contractor,  refine  the  risk- 
monitoring  plans  and  procedures. 

•  Establish  the  program  reporting  requirements 
with  the  contractor.  Describe  the  risk  man¬ 
agement  information  system  that  the  program 
has  established,  including  procedures  for  pro¬ 
viding  information  for  data  entry,  and  iden¬ 
tify  reports  for  the  PMO  and  contractor. 

•  In  conjunction  with  the  contractor,  identify 
other  risk-management  activities  that  need  to 
be  performed. 


•  Manage  the  program  risk  in  accordance  with 
the  risk  management  plan  and  contract. 

•  Working  with  the  contractor,  refine  the  risk¬ 
monitoring  plans  and  procedures  and  de¬ 
velop  appropriate  measures  and  metrics  to 
track  moderate-  and  high-risk  items. 

4.8  RISK  MANAGEMENT 
REPORTING  AND 
INFORMATION  SYSTEM 

The  PMO  should  have  a  practical  method  for 
risk-management  reporting,  and  an  information 
system  that  supports  a  risk  management  program. 
The  reporting  needs  of  the  PM  establish  the  type, 
format,  and  frequency  of  information  sharing. 
The  IPT  concept  suggests  that  the  entire  acqui¬ 
sition  program  team  needs  access  to  the  risk  man¬ 
agement  information,  and  the  prime  contractor(s) 
should  have  access  to  information,  consistent 
with  acquisition  regulations.  The  reporting  and 
information  system  chosen  may  be  Govemment- 
or  contractor-owned.  See  Chapter  5  for  an  ex¬ 
ample  of  an  MIS. 

4.9  RISK  MANAGEMENT  TRAINING 

A  successful  management  program  depends,  to 
a  large  extent,  on  the  level  of  risk  management 
training  the  PMO  members  and  the  functional 
area  experts  receive.  The  training  will  prepare 
them  for  critical  tasks,  such  as  risk  assessments. 
DoD  schools  offer  some  risk-management  train¬ 
ing;  however,  PMs  will  need  to  organize  and 
conduct  principal  training  for  the  program  of¬ 
fice.  A  three-part  framework  for  training  cov¬ 
ers  program-specific  risk  management  issues, 
general  structure  and  process,  and  techniques. 

( 1 )  The  program-  specific  training  should  ensure 
that  everyone  has  a  common  vision.  It 
should  cover  the  acquisition  strategy,  the 
companion  risk  management  plan,  the  PM’s 
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risk-management  structure  and  associated 
responsibilities,  and  the  MIS. 

(2)  The  following  topics  provide  a  starting 
point  for  general  training  syllabus  devel¬ 
opment.  The  final  syllabus  should  be  tai¬ 
lored  to  meet  the  program’s  specific  needs. 
Table  4-3  provides  a  list  of  references  that 
will  be  useful  in  developing  the  syllabus 
and  lesson  plans. 

•  Concept  of  Risk, 

•  Risk  Planning, 

•  Risk  Identification, 

•  Risk  Analysis  (as  applicable), 

•  Risk  Handling,  and 


•  Risk  Monitoring. 

(3)  The  third  area  of  training  concerns  risk- 
management  techniques,  concentrating  on 
the  techniques  the  PMO  plans  to  employ. 
The  training  should  focus  on  how  to  use 
the  techniques  and  should  include  ex¬ 
amples  of  their  use.  Chapter  5,  Risk  Man¬ 
agement  Techniques,  of  this  Guide  provides 
a  starting  point.  It  contains  a  general  dis¬ 
cussion  of  a  set  of  techniques  that  address 
all  elements  of  the  risk  management  pro¬ 
cess.  The  discussion  of  each  technique  con¬ 
tains  a  list  of  references  that  provide  a  more 
in-depth  description  of  the  technique.  The 
set  of  techniques  is  not  exhaustive  and  the 
program  office  should  add  to  the  list,  if 
necessary. 
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Document 

Description 

DoD  4245. 7-M,  Transition  from  Development 
to  Production ,  September  1985. 

Provides  a  structure  for  identifying  technical  risk 
areas  in  the  transition  from  a  program’s  development 
to  production  phases.  The  structure  is  geared  toward 
development  programs  but,  with  modifications,  could 
be  used  for  any  acquisition  program.  The  structure 
identifies  a  series  of  templates  for  each  of  the 
development  contractor’s  critical  engineering 
processes.  The  template  includes  potential  areas  of 
risk  and  methods  for  reducing  risk  in  each  area. 

Risk  Management  Concepts  and  Guidance, 
Defense  Systems  Management  College, 

March  1989.  (Superseded  by  this  Risk 
Management  Guide.) 

Devoted  to  various  aspects  of  risk  management. 

Systems  Engineering  Management  Guide, 
Defense  Acquisition  University  Press, 

January  2001,  Section  15. 

Devoted  to  risk  analysis  and  management  and 
provides  a  good  overview  of  the  risk  management 
process. 

Continuous  Risk  Management  Guide, 

Software  Engineering  Institute,  Carnegie 

Mellon  University,  1996. 

Provides  a  risk  management  methodology  similar  to 
the  one  described  in  the  Defense  Acquisition  Deskbook. 
Its  value  is  that  it  subdivides  each  process  into  a  series 
of  steps;  this  provides  useful  insights.  Appendix  A 
describes  40  risk-management  techniques,  the  majority 
of  which  are  standard  management  techniques  adapted 
to  risk  management.  This  makes  them  a  useful 
supplement  to  the  Defense  Acquisition  Deskbook 
identified  techniques. 

A  Systems  Engineering  Capability  Maturity 
Model,  Version  1.0  Software  Engineering 
Institute  (Carnegie  Mellon  University), 
Handbook  SECMM-94-04,  December  1994. 

Describes  one  approach  to  conducting  an  Industry 
Capabilities  Review.  Section  PA  10  (pp.  4-72-4-76) 
discusses  software  risk  management.  The  material 
presented  in  this  handbook  also  can  be  tailored  to 
apply  to  system  and  hardware  risk. 

A  Software  Engineering  Capability  Maturity 
Model,  Version  1 .01  Software  Engineering 
Institute  (Carnegie  Mellon  University), 

Technical  Report,  December  1996. 

Describes  an  approach  to  assess  the  software 
acquisition  processes  of  the  acquiring  organization 
and  identifies  areas  for  improvement. 

Capability  Maturity  Model  for  Software 
(SM-CMM),  Version  1.1./CMU/SEI-93-TR-24, 
February  1993. 

This  is  a  tool  that  allows  an  acquiring  organization  to 
assess  the  software  capability  maturity  of  an 
organization. 

Taxonomy-Based  Risk  Identification, 

Software  Engineering  Institute,  Carnegie 
Mellon  University,  CMU/SEI-93-TR-6 
(ESC-TR-93-1 83,  June  1993. 

Describes  a  method  for  facilitating  the  systematic  and 
repeatable  identification  of  risks  associated  with  the 
development  of  a  software-intensive  project.  This 
method  has  been  tested  in  active  Government-funded 
defense  and  civilian  software  development  projects. 

The  report  includes  macro-level  lessons  learned  from 
the  field  tests. 

NAVSO  P-6071. 

Navy  “best  practices”  document  with  recommended 
implementations  and  further  discussion  on  the 
material  in  DoD  4245. 7-M. 

Table  4-3.  Risk  Management  Reference  Documents 
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Document 

Description 

Risk  Management,  AFMC  Pamphlet  63-1 01 , 
July  1997. 

An  excellent  pamphlet  on  risk  management  that  is 
intended  to  provide  PMs  and  the  PMO  with  a  basic 
understanding  of  the  terms,  definitions,  and  processes 
associated  with  effective  risk  management.  It  is  very 
strong  on  how  to  perform  pre-contract  award  risk 
management. 

Defense  Acquisition  Deskbook 

Primary  reference  tool  for  defense  acquisition  work 
force;  contains  over  1 ,000  mandatory  and 
discretionary  publications  and  documents  which 
promulgate  acquisition  policy  and  guidance.  Part  of  the 
AT&L  Knowledge  Sharing  System  (AKSS)  (http:// 
deskbook.dau.mil/jsp/default.jsp). 

Acquisition  Software  Development 

Capability  Evaluation,  AFMC  Pamphlet 

63-103, 15  June  94. 

Describes  one  approach  to  conducting  an  Industry 
Capabilities  Review.  This  two-volume  pamphlet  was 
generated  from  material  originated  at  Aeronautical 
Systems  Center.  The  concepts  support  evaluations 
during  source  selection  and  when  requested  by  IPTs. 
The  material  presented  in  this  pamphlet  also  can  be 
tailored  to  apply  to  system  and  hardware  risk 
management. 

Risk  Management  Critical  Process 
Assessment  Tool,  Air  Force  SMC/AXD, 

Version  2,  9  June  1 998. 

Provides  guidance  and  extensive  examples  for 
developing  RFP  Sections  “L”  and  “M,”  plus  source 
selection  standards  or  risk  management.  Also  includes 
technical  evaluation  and  review  questions,  which  are 
helpful  for  assessing  a  risk  management  process;  and 
risk  trigger  questions,  which  are  helpful  for  risk 
identification. 

NAVSO  P-3686,  Top  Eleven  Ways  to 

Manage  Technical  Risk,  October  1 998. 

Contains  Navy  approach  to  risk  management  with 
baseline  information,  explanations,  and  best  practices 
that  contribute  to  a  well-founded  technical  risk 
management  program. 

Risk  Focus  Area  of  the  Program 

Management  Community  of  Practice 
(http://www.pmcop.dau.mil) 

Provides  comprehensive  and  ready  source  of  current 
tools,  papers,  and  practices  in  risk  management  field. 

Table  4-3.  Risk  Management  Reference  Documents 
(continued) 
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5 

RISK  MANAGEMENT 
TECHNIQUES 


5.1  INTRODUCTION 

This  Chapter  provides  top-level  information  on 
a  number  of  techniques  currently  used  in  DoD, 
and  a  combination  of  techniques  used  by  the  Ser¬ 
vices,  industry,  and  academia.  Collectively,  they 
focus  on  the  components  of  the  risk  manage¬ 
ment  process  and  address  critical  risk  areas  and 
processes.  The  write-ups  describe  the  techniques 
and  give  information  on  their  application  and 
utility.  The  descriptions  are  at  a  level  of  detail 
that  should  permit  potential  users  to  evaluate  the 
suitability  of  the  techniques  for  addressing  their 
needs;  however,  the  material  does  not,  in  most 
cases,  provide  all  the  information  that  is  required 
to  use  a  technique.  Readers  will  find  that  if  a 
particular  technique  looks  promising,  they  can 
obtain  enough  information  from  the  references 
and  tools  that  will  enable  program  offices  to  ap¬ 
ply  them.  The  descriptions  are  in  a  format  that 
aids  comparison  with  other  approaches. 

5.2  OVERVIEW 

Techniques  are  available  to  support  risk  man¬ 
agement  activities.  None  are  required  by  DoD, 
but  some  have  been  successfully  used  in  the 
past  by  DoD  PMs.  Many  of  the  techniques 
support  processes  that  are  part  of  sound  man¬ 
agement  and  systems  engineering  and  give 
Government  and  contractor  PMs  the  tools  for 
considering  risk  when  making  decisions  on 
managing  the  program. 


Several  tools  have  been  developed  to  support 
each  of  the  components  of  the  risk  management 
process,  i.e.,  planning,  assessing,  handling,  and 
monitoring  and  documenting.  Although  tool 
developers  may  claim  otherwise,  none  are 
integrated  to  totally  satisfy  all  needs  of  a  PM. 
Most  likely,  a  PM  will  choose  an  overall  risk 
strategy,  write  a  plan  to  reflect  his  strategy,  re¬ 
view  the  list  of  proven  techniques  to  support 
the  components  of  risk  management,  assess  the 
techniques  against  the  program’s  needs  and 
available  resources,  tailor  the  techniques  to  suit 
the  needs  of  the  program,  and  train  program 
office  members  to  implement  the  plan. 

5.3  RISK  PLANNING  TECHNIQUES 

5.3.1  Description 

This  technique  suggests  an  approach  to  risk  plan¬ 
ning;  the  process  of  developing  and  document¬ 
ing  an  organized,  comprehensive  approach.  It 
also  suggests  interactive  strategy  and  methods 
for  identifying  and  tracking  risk  drivers,  devel¬ 
oping  risk-handling  plans,  performing  continu¬ 
ous  assessments  to  determine  how  risks  have 
changed,  and  planning  adequate  resources.  The 
risk  planning  technique  is  applicable  to  all  func¬ 
tional  areas  in  the  program,  especially  critical  ar¬ 
eas  and  processes.  Using  the  acquisition  strat¬ 
egy  as  a  starting  point  results  in  the  development 
of  a  program  risk  management  strategy,  from 
which  flows  a  management  plan  that  provides 
the  detailed  information  and  direction  necessary 
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to  conduct  an  effective  management  program. 
This  risk  management  plan  provides  the  PM  with 
an  effective  method  to  define  a  program,  one 
that  fixes  responsibility  for  the  implementation 
of  its  various  aspects,  and  supports  the 
acquisition  strategy. 

The  technique  should  first  be  used  in  the  Con¬ 
cept  Refinement  (CR)  Phase  in  conjunction  with 
the  development  of  the  initial  Technology  De¬ 
velopment  Strategy  (TDS).  Subsequently,  it  may 
be  used  to  update  the  management  plan  on  the 
following  occasions:  (1)  whenever  the  acquisi¬ 
tion  strategy  changes,  or  there  is  a  major  change 
in  program  emphasis;  (2)  in  preparation  for  major 
decision  points;  (3)  in  preparation  for  and 
immediately  following  technical  audits  and 
reviews;  (4)  concurrent  with  the  review  and 


update  of  other  program  plans;  and  (5)  in 
preparation  for  a  PMO  submission. 

The  PMO  risk  management  coordinator,  if 
assigned,  develops  the  risk  management  plan 
based  on  guidance  provided  by  the  PM,  and 
coordinating  with  the  Program  Level  IPT.  To 
be  effective,  the  PM  must  make  risk  manage¬ 
ment  an  important  program  management  func¬ 
tion  and  must  be  actively  involved  in  the  risk 
planning  effort.  Planning  requires  the  active  par¬ 
ticipation  of  essentially  the  entire  PMO  and 
contractor  team. 

5.3.2  Procedures 

Figure  5-1  graphically  depicts  the  process  to 
be  followed  in  applying  this  technique.  The 


Input 

•  Acquisition  strategy 

•  Prior  risk 
management  plan 
(if  any) 

•  Known  risks 

•  System  description 

•  Program  description 

•  Key  ground  rules  and 
assumptions 


PM  Guidance 

I 

•  Evaluate  risk  planning 
requirements 

•  Evaluate  the  program’s  current 
risk  situation 

•  Develop  a  risk  management 
strategy 

•  Determine  the  tasks  and 
guidance  required  to  implement 

►  the  risk  management  strategy 

•  Develop  the  PMO’s  approach  to 
risk  management  in  general 

•  Provide  application  guidance  for 
risk  management  component 
processes 

•  Develop  inputs  for  other 
acquisition  strategies  and 
program  processes 

•  Program-Level  IPT  (or  equivalent 

such  as  Risk  Management  Board) 

•  Risk  management  coordinator 


Output 

Risk  Management 
Plan 

Risk  Management 
Training 


Figure  5-1.  Risk  Planning  Technique  Input  and  Output 
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procedure  consists  of  a  number  of  iterative  activi¬ 
ties  that  result  in  the  development  of  the  risk  man¬ 
agement  strategy  and  a  Risk  Management  Plan. 

The  acquisition  strategy  and  related  manage¬ 
ment  planning  efforts  (program  management, 
and  systems  engineering),  program  constraints, 
and  any  existing  risk  management  planning  are 
integrated  and  evaluated  in  the  context  of  the 
PM’s  guidance,  which  provides  the  direction 
for  the  planning  process.  Typical  types  of  PM 
guidance  are  concerns  about  certain  categories 
of  risk,  guidance  on  funding  of  handling 
activities,  emphasis  to  be  placed  on  risk  man¬ 
agement  training,  and  frequency  and  type  of 
internal  reports. 

The  integration  and  evaluation  of  the  primary 
inputs  establish  the  requirements  and  scope  of 
the  planning  effort  through  an  assessment  of 
the  program’s  current  risk  situation.  The  results 
of  the  assessment  provide  the  basis  for  devel¬ 
opment  of  management  strategy.  The  strategy 
should  reflect  the  level  of  risk  that  the  PM  is 
prepared  to  accept,  and  should  provide  guid¬ 
ance  on  how  and  when  known  risks  will  be 
reduced  to  acceptable  levels.  It  should  also 
describe  the  risk  management  process  the  PMO 
will  employ  and  the  organization  and  structure 
of  the  management  program,  addressing  things 
such  as  risk  ratings,  the  use  of  an  MIS,  policy 
and  procedures  on  sharing  risk  management 
information,  and  training. 

The  PMO  should  create  an  MIS  early  in  the 
planning  process.  It  will  serve  as  a  planning 
source  and  the  data  may  be  used  for  creating 
reports.  It  will  also  become  the  repository  for 
all  current  and  historical  information  related  to 
risk.  Eventually,  this  information  may  include 
risk  assessment  documents,  contract  deliverables, 
if  appropriate,  and  other  risk-related  reports. 

Based  on  the  management  strategy,  the  plan  iden¬ 
tifies  specific  tasks  to  be  accomplished  and 


assigns  responsibility  for  their  execution.  The 
timing  of  these  tasks  should  be  incorporated  into 
an  integrated  critical  path  master  schedule  or 
equivalent.  Guidance  for  task  execution  and  con¬ 
trol  should  also  be  developed,  covering  such 
things  as  the  suggested  techniques  to  be  used 
for  each  component,  any  assistance  available  to 
Sub-Tier  IPTs,  the  use  of  funds,  the  policy  on 
the  use  of  independent  risk  assessors,  etc.  This 
information  may  be  documented  in  a  risk  man¬ 
agement  plan.  A  sample  format  is  shown  in  Figure 
5-2.  Appendix  B  contains  two  examples  of  a 
Risk  Management  Plan. 

The  contents  of  the  risk  management  strategy 
and  plan  should  be  consistent  with  the  acquisi¬ 
tion  strategy  and  other  program  plans  derived 
from  the  acquisition  strategy.  Hence,  it  should 
be  tailored  to  each  program  rather  than  attempt¬ 
ing  to  use  the  same  process  and  its  implementa¬ 
tion  on  all  programs.  This  will  help  to  ensure  that 
risk  is  considered  in  all  program  activities  and 
that  it  does  not  become  a  “stove  pipe”  function. 

5.4  RISK  ASSESSMENT  TECHNIQUES 

5.4.1  Product  (WBS)  Risk  Assessment 

5.4.1. 1  Description.  This  technique  identifies 
those  risks  associated  with  a  given  system  con¬ 
cept  and  design.  The  difference  between  the  pro¬ 
cess  (DoD  4245. 7-M)  technique  and  this  ap¬ 
proach  is  that  DoD  4245. 7-M  addresses  the 
contractor’s  engineering  and  manufacturing  pro¬ 
cesses  and  this  technique  focuses  on  the  result¬ 
ing  product.  This  technique  is  used  to  identify 
and  analyze  risks  in  the  following  critical  risk 
areas:  design  and  engineering,  technology,  lo¬ 
gistics,  production,  concurrency,  plus  others  as 
needed  for  both  hardware  and  software. 

The  WBS  is  the  starting  point  to  describe  con¬ 
tract  work  to  be  done  and  the  resulting  product 
and  is  the  basis  for  determining  risk  events  in 
each  critical  risk  area.  The  risk  events — events 
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INTRODUCTION.  This  section  should  address  the  purpose  and  objective  of  the  plan,  and  provide  a  brief 
summary  of  the  program,  to  include  the  approach  being  used  to  manage  the  program,  and  the  acquisition 
strategy. 

PROGRAM  SUMMARY.  This  section  contains  a  brief  description  of  the  program,  including  the  acquisition 
strategy  and  the  program  management  approach.  The  acquisition  strategy  should  address  its  linkage  to  the 
risk  management  strategy. 

DEFINITIONS.  Definitions  used  by  the  program  office  should  be  consistent  with  DoD  definitions  for  ease  of 
understanding  and  consistency.  However,  the  DoD  definitions  allow  program  managers  flexibility  in  constructing 
their  risk  management  programs.  Therefore,  each  program’s  risk  management  plan  may  include  definitions 
that  expand  the  DoD  definitions  to  fit  its  particular  needs.  For  example,  each  plan  should  include,  among  other 
things,  definitions  for  the  ratings  used  for  technical,  schedule,  and  cost  risk. 

RISK  MANAGEMENT  STRATEGY  AND  APPROACH.  Provide  an  overview  of  the  risk  management  approach, 
to  include  the  status  of  the  risk  management  effort  to  date,  and  a  description  of  the  program  risk  management 
strategy. 

ORGANIZATION.  Describe  the  risk  management  organization  of  the  program  office  and  list  the  responsibilities 
of  each  of  the  risk  management  participants. 

RISK  MANAGEMENT  PROCESS  AND  PROCEDURES.  Describe  the  program  risk  management  process  to  be 
employed,  i.e.,  risk  planning,  assessment,  handling,  monitoring  and  documentation,  and  a  basic  explanation 
of  these  components.  Also  provide  guidance  for  each  of  the  risk  management  steps  in  the  process.  If  possible, 
the  guidance  should  be  as  general  as  possible  to  allow  the  program’s  risk  management  organization  (e.g., 
IPTs)  flexibility  in  managing  the  program  risk,  yet  specific  enough  to  ensure  a  common  and  coordinated 
approach  to  risk  management.  It  should  address  how  the  information  associated  with  each  element  of  the  risk 
management  process  will  be  documented  and  made  available  to  all  participants  in  the  process,  and  how  risks 
will  be  tracked,  to  include  the  identification  of  specific  metrics  if  possible. 

RISK  PLANNING.  This  section  describes  the  risk  planning  process  and  provides  guidance  on  how  it  will  be 
accomplished,  and  the  relationship  between  continuous  risk  planning  and  this  RMP.  Guidance  on  updates  of 
the  RMP  and  the  approval  process  to  be  followed  should  also  be  included. 

RISK  ASSESSMENT.  This  section  of  the  plan  describes  the  assessment  (identification  and  analysis)  process. 
It  includes  procedures  for  examining  the  critical  risk  areas  and  processes  to  identify  and  document  the  associated 
risks.  It  also  summarizes  the  analyses  process  for  each  of  the  risk  areas  leading  to  the  determination  of  a  risk 
rating.  This  rating  is  a  reflection  of  the  potential  impact  of  the  risk  in  terms  of  its  variance  from  known  Best 
Practices  or  probability  of  occurrence,  its  consequence,  and  its  relationship  to  other  risk  areas  or  processes. 
This  section  may  include: 

•  Overview  and  scope  of  the  assessment  process 

•  Sources  of  information 

•  Information  to  be  reported  and  formats 

•  Description  of  how  risk  information  is  retained 

•  Assessment  techniques  and  tools. 

RISK  HANDLING.  This  section  describes  the  risk-handling  options,  and  identifies  tools  that  can  assist  in 
implementing  the  risk-handling  process.  It  also  provides  guidance  on  the  use  of  the  various  handling  options 
for  specific  risks. 

RISK  MONITORING.  This  section  describes  the  process  and  procedures  that  will  be  followed  to  monitor  the 
status  of  the  various  risk  events  identified.  It  should  provide  criteria  for  the  selection  of  risks  to  be  reported  on, 
and  the  frequency  of  reporting.  Guidance  on  the  selection  of  metrics  should  also  be  included. 

RISK  MANAGEMENT  INFORMATION  SYSTEM,  DOCUMENTATION  AND  REPORTS.  This  section  describes 
the  MIS  structure,  rules,  and  procedures  that  will  be  used  to  document  the  results  of  the  risk  management 
process.  It  also  identifies  the  risk  management  documentation  and  reports  that  will  be  prepared;  specifies  the 
format  and  frequency  of  the  reports;  and  assigns  responsibility  for  their  preparation. 


Figure  5-2.  Sample  Format  for  Risk  Management  Plan 
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that  might  have  a  detrimental  impact  on  the 
system,  subsystems,  or  components — are  evalu¬ 
ated  to  identify  and  characterize  specific  risks 
ratings  and  prioritization. 

This  technique  should  be  used  shortly  after  the 
completion  of  the  prime  contractor’s  WBS. 
Thereafter,  it  should  be  used  regularly  up  to  the 
start  of  production.  The  technique  can  be  used 
independently  or  in  conjunction  with  other  risk 
assessment  techniques,  such  as  the  Process 
(DoD  4245. 7-M)  Risk  Assessment  technique.  It 
may,  if  appropriate,  also  be  used  in  conjunction 
with  the  Integrated  Baseline  Review  (IBR), 
which  is  conducted  within  6  months  of  contract 
award.  A  website  is  also  available  at  http:// 
www.acq.osd.mil/pm/ibrmats/ibrmats.htm, 
which  discusses  the  IBR  Process. 

To  apply  this  technique,  joint  Government  and 
industry  evaluation  teams  should  examine  the 
appropriate  WBS  levels  in  each  Sub-Tier  IPTs 
product  area.  If  necessary,  complementary 
industry-only  teams  may  take  an  in-depth  look 
at  selected  areas  at  lower  WBS  levels.  At  times, 
it  may  be  desirable  to  include  outside  industry 


experts  on  the  teams  to  aid  in  the  examination  of 
specific  WBS  elements  or  functional  areas. 

5.4.1.2  Procedures.  Figure  5-3  depicts  the  pro¬ 
cess  used  in  this  technique.  The  first  step  is  to 
review  the  WBS  elements  down  to  the  level  be¬ 
ing  considered,  and  identify  risk  events.  This 
review  should  consider  the  critical  areas  (de¬ 
sign  and  engineering,  technology,  logistics,  etc.) 
that  may  help  to  describe  risk  events.  Table  5-1 
shows  a  partial  listing  of  these  elements. 

Using  information  from  a  variety  of  sources, 
such  as  program  plans,  prior  risk  assessments, 
expert  interviews,  etc.,  the  WBS  elements  are 
examined  to  identify  specific  risks  in  each  criti¬ 
cal  area.  The  risk  event,  are  then  analyzed  to 
determine  probability  of  occurrence  and  conse¬ 
quences/impacts,  along  with  any  interdependen¬ 
cies  and  risk  event  priorities.  Several  techniques 
and  tools  are  available  to  accomplish  this,  in¬ 
cluding,  among  others,  technology  assessments, 
modeling  and  simulation,  hazard  analysis,  and 
fault  tree  analysis. 


Input 

•  Program  Plans 

•  Past  Projected  Data 

•  WBS 

•  Integrated  Master  Schedule 
(or  equivalent) 

•  Critical  Area  Evaluation  Criteria 

1 

Output 

•  Examine  WBS  elements  and 

•  Risk  Information  Forms 

•  Lesson  Learned 

identify  risk  events 

•  Prioritized  List  of  Risks 

•  Expert  Interview  Data  — ► 

•  Analyze  risk  events 

^  •  List  of  Aggregated 

•  Test  Results 

(Includes  rating  and 

Risks 

•  Integrated  Baseline 

prioritizing  risk  events) 

•  Watch  Lists 

Review 

f 

1 

•  Sub-Tier  IPT  Evaluation  Teams 

•  “Outside”  Industrial  Experts 

Figure  5-3.  Product  (WBS)  Risk  Assessment  Technique  Input  and  Output 
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Critical  Risk 
Areas 

Example  Elements 

Design  and 
Engineering 

•  Design/technology  approach 

•  Operational  environments 

•  External/internal  interfaces 

•  Use  of  standard  parts/program 
parts  list 

•  System/subsystem  critical  design 
requirement 

•  Integration  requirements 

•  Human-machine  interface 

•  Design  growth  capacity 

•  Design  maturity 

•  Safety  and  health  hazards 

•  Manpower,  training  and  skill  profiles 

Logistics 

•  Operations  and  Maintenance 
(O&M)  concept 

•  System  diagnostic  requirement 

•  Repairability  and  Maintainability 
(R&M)  requirements 

•  Supply  support  requirements 

•  Built-in  Test  (BIT)  requirements 

•  Support  equipment  requirements 

•  Maintenance  interfaces 

•  Level  of  repair  decisions 

•  Training  equipment  design 

Testing 

•  Integrated  test 

•  Qualification  testing 

•  Subsystem  test  limits 

•  Test  environmental  acceleration 

•  Supportability  test  results 

Manufacturing 

•  Design  producibility 

•  Manufacturing  capability 
requirements 

•  Parts/assemblies  availability 

•  Special  tooling/test  equipment  planning 
personnel  availability 

•  Process/tooling  proofing 

•  Production  equipment  availability 

Concurrency 

•  Program  schedule  adequacy 

•  Development  phases  concurrency 

Table  5-1.  Critical  Risk  Areas  and  Example  Elements 


The  results  of  this  analysis  should  be  documented 
in  a  program- specific  standard  format,  such  as  a 
Risk  Information  Form  (RIF).  The  risks,  along 
with  others  identified  using  other  techniques,  can 
be  prioritized  and  aggregated  using  the  technique 
described  later  in  this  chapter. 

5.4.2  Process  (DoD  4245.7-M) 

Risk  Assessment 

5.4.2.1  Description.  This  technique  is  used 
to  assess  (identify  and  analyze)  program  tech¬ 
nical  risks  resulting  from  the  contractor’s  pro¬ 
cesses.  It  is  based  on  the  application  of  the 
technical  risk  area  templates  found  in  DoD 
4245.7-M.  These  templates  describe  the  risk 


areas  contained  in  the  various  technical  pro¬ 
cesses  (e.g.,  design,  test,  production,  etc.)  and 
specify  methods  for  reducing  risks  in  each 
area.  Success  of  any  risk  reduction  efforts  as¬ 
sociated  with  this  technique  will  depend  on 
the  contractor’s  ability  and  willingness  to  make 
a  concerted  effort  to  replace  any  deficient  en¬ 
gineering  practices  and  procedures  with  best 
industrial  practices. 

One  of  the  primary  benefits  of  this  technique  is 
that  it  addresses  pervasive  and  important 
sources  of  risk  in  most  DoD  acquisition  pro¬ 
grams  and  uses  fundamental  engineering  prin¬ 
ciples  and  proven  procedures  to  reduce  techni¬ 
cal  risks.  The  technique  is  accepted  by  many 
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aerospace  companies  in  normal  business  activi¬ 
ties,  and  in  fact,  was  developed  by  a  group  of 
Government  and  aerospace  experts. 

The  technique  is  primarily  applicable  during  the 
Technology  Development  (TD)  Phase,  and  the 
System  Demonstration  part  of  the  System  De¬ 
velopment  and  Demonstration  (SDD)  Phase 
of  program  development.  In  the  TD  Phase  it 
provides  a  detailed  checklist  of  processes  that 
the  contractor  needs  to  address;  in  the  System 
Demonstration  part  of  the  SDD  Phase,  the  pro¬ 
cesses  are  being  implemented  in  preparation  for 
Low  Rate  Initial  Production  (LRIP).  The  de¬ 
scription  of  each  template  in  DoD  4245. 7-M 
shows  the  phases  in  which  the  template  should 
be  applied.  The  specific  timing  of  the  applica¬ 
tion  within  the  phases  should  be  determined 
based  on  the  type  of  program,  the  acquisition 
strategy  and  plans,  and  the  judgment  of  pro¬ 
gram  officials.  It  should  also  be  used  in 


preparation  for  milestone  decisions  and  when 
preparing  for  source  selection.  This  technique 
may  be  used  independently  or  in  conjunction 
with  other  risk  assessment  techniques.  When 
feasible,  a  Government-industry  evaluation 
team  should  be  formed  early  in  the  program  to 
apply  this  technique. 

5.4.2.2  Procedures.  Figure  5-4  shows  the  ba¬ 
sic  approach  used  in  this  technique.  The  DoD 
4245. 7-M  templates  are  used  in  conjunction 
with  the  contract  requirements  and  specifica¬ 
tions  to  identify  those  technical  processes  criti¬ 
cal  to  the  program  and  to  establish  a  program 
baseline  of  contractor  processes.  When  pos¬ 
sible,  the  program  baseline  should  be  determined 
by  evaluating  actual  contractor  performance,  as 
opposed  to  stated  policy.  For  example,  design 
policy  should  be  determined  from  interviewing 
designers  and  not  simply  from  reviewing  writ¬ 
ten  corporate  policies. 
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•  DoD  4245.7-M  Templates 

•  Corporate  Policies,  Practices 
&  Procedures 

•  Contract  Requirements 
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•  Identify  Program’s  Critical 
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•  Technical  Baseline 
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•  Past  Project  Data 

•  Develop  Program  Baseline 

•  Technical  Risk 
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Figure  5-4.  Process  (DoD  4245.7-M)  Risk  Assessment  Technique  Input  and  Output 
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This  program  baseline  should  then  be  compared 
to  a  baseline  of  industry-wide  processes  and 
practices  that  are  critical  to  the  program.  The 
baseline  should  be  developed  by  reviewing  and 
compiling  known  best  practices  in  use  by  vari¬ 
ous  companies  in  both  defense  and  non-defense 
sectors.  One  source  of  best  practices  informa¬ 
tion  is  the  Program  Manager’s  Work  Station 
(PMWS),  a  series  of  PC  expert  systems 
designed  to  aid  in  the  implementation  of  DoD 
4245. 7-M.  The  point  of  contact  for  the  PMWS 
is  the  Best  Manufacturing  Practices  Center  of 
Excellence  (http://www.bmpcoe.org). 

The  differences  between  the  two  baselines  are  a 
reflection  of  the  technical  process  risk  present. 
These  results  should  be  documented  in  a  stan¬ 
dard  format,  such  as  a  program- specific  Risk  In¬ 
formation  Form  (see  MIS  discussion  this  section) 
to  facilitate  the  development  of  a  risk  handling 
and  risk  reporting  plan. 

5.4.3  Program  Documentation 

Evaluation  Risk  Identification 

5.4.3.1  Description.  This  technique  provides  a 
methodology  for  comparing  key  program  docu¬ 
ments  and  plans  to  ensure  that  they  are  consis¬ 
tent  and  traceable  to  one  another.  Program 
documents  and  plans  are  hierarchical  in  nature. 
If  the  contents  (activities,  events,  schedules, 


requirements,  specifications,  etc.)  of  a  document 
or  plan  do  not  flow  from  or  support  the  contents 
of  those  above,  below,  or  adjacent  to  it,  there  is 
a  strong  chance  that  risk  will  be  introduced  into 
the  program  or  that  known  risks  will  not  be  ad¬ 
equately  addressed.  This  technique  reduces  those 
risks  and  improves  the  quality  of  program 
documentation. 

This  technique  can  be  used  in  any  acquisition 
phase  as  documents  or  plans  are  being  devel¬ 
oped  or  updated.  The  comparison  of  program 
documentation  and  plans  should  be  performed 
by  a  small  team  of  experienced,  knowledgeable 
personnel  who  are  intimately  familiar  with  the 
total  program. 

5.4.3.2  Procedures.  Figure  5-5  shows  the  pro¬ 
cess  used  in  this  technique.  The  primary  inputs 
to  the  process  are  the  PMO  documents  that  de¬ 
tail  the  steps  involved  in  executing  the  program. 
These  include,  for  example,  the  Mission  Need 
Statement  (MNS),  Operational  Requirements 
Document  (ORD),  acquisition  plan,  any  master 
management  plan,  Test  and  Evaluation  Master 
Plan  (TEMP),  manufacturing  plan,  etc.  The 
MNS  is  being  replaced  by  the  Initial  Capabili¬ 
ties  Document  (ICD),  and  the  ORD  is  being  re¬ 
placed  by  the  Capability  Development  Docu¬ 
ment  (CDD).  Another  set  of  key  input  documents 
are  those  used  to  communicate  with  the  prime 
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Figure  5-5.  Plan  Evaluation  Technique  Input  and  Output 
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contractor,  e.g.,  WBS,  specifications,  Statement 
of  Work  (SOW)  or  equivalent  such  as,  Statement 
of  Objectives,  etc.  Before  any  comparison,  the 
PMO  should  review  all  documents  for  accuracy 
and  completeness.  Figure  5-6  shows  an  example 
of  the  type  of  correlation  that  should  exist  among 
the  MNS,  ORD,  and  TEMP  during  the  CR  and 
TD  Phases. 

If  the  comparison  shows  any  gaps  or 
inconsistencies,  reviewers  should  identify  them 
as  possible  risks  on  a  RIF,  the  output  of  this  pro¬ 
cess. 

5.4.4  Threat  and  Requirements 
Risk  Assessment 

5.4.4. 1  Description.  This  technique  describes 
an  approach  to  assess  risks  associated  with  re¬ 
quirements  and  threat  and  to  identify  require¬ 
ments  and  threat  elements  that  are  risk  drivers. 
Because  operational  needs,  environmental  de¬ 
mands,  and  threat  determine  system  performance 
requirements,  to  a  large  degree,  they  are  a  major 
factor  in  driving  the  design  of  the  system  and 


can  introduce  risk  in  a  program.  Further,  with 
the  introduction  of  CAIV,  PMs  and  users  are 
directed  to  examine  performance  requirements 
and  identify  areas  that  are  not  critical  and  are 
available  for  trade  to  meet  cost  objectives.  Risk 
is  a  factor  in  CAIV  considerations. 

The  requirements  risk  assessment  process  focuses 
on:  determining  if  operational  requirements  are 
properly  established  and  clearly  stated  for  each 
program  phase;  ensuring  that  requirements  are 
stable  and  the  operating  environment  is  ad¬ 
equately  described;  addressing  logistics  and  suit¬ 
ability  needs;  and  determining  if  requirements 
are  too  constrictive,  thereby  identifying  a  spe¬ 
cific  solution.  The  evaluation  of  the  threat  risk 
assessment  process’  maturity  addresses: 
uncertainty  in  threat  accuracy  and  stability, 
sensitivity  of  design  and  technology  to  threat, 
vulnerability  of  the  system  to  threat  countermea¬ 
sures,  and  vulnerability  of  the  program  to  intelli¬ 
gence  penetration.  PMs  should  view  require¬ 
ments  in  the  context  of  the  threat  and  accurately 
reflect  operational,  environmental,  and  suitabil¬ 
ity  requirements  in  design  documents. 


Figure  5-6.  Concept  Refinement  (CR)  andTechnology  Development  (TD)  Phases 
Correlation  of  Selected  Documents  (Example) 
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PMs  should  use  threat  and  requirements  assess¬ 
ments  during  the  early  phases  of  program  de¬ 
velopment  and,  as  necessary,  as  the  program  ad¬ 
vances  through  development.  Early  and  com¬ 
plete  understanding  of  the  requirements  and 
threat  precludes  misunderstandings  between  the 
requirements  and  development  communities, 
helps  to  identify  risk  areas,  and  allows  early 
planning  to  handle  risk.  Consequently,  the  user 
should  be  actively  involved  in  this  process  from 
the  beginning. 

5.4.4.2  Procedures.  Figure  5-7  depicts  the  pro¬ 
cess  used  in  this  technique.  The  basic  approach 
is  to  conduct  a  thorough  review  of  the  documents 
containing  performance  requirements  and  threat 
information,  e.g.,  ORD,  TEMP,  System  Speci¬ 
fication,  System  Threat  Assessment  (STA),  De¬ 
sign  Reference  Mission  Profile,  etc.,  to  deter¬ 
mine  stability,  accuracy,  operating  environment, 
logistics  and  suitability  requirements,  and  con¬ 
sistency  between  these  requirements  and  the 
threat  considerations  cited  above.  There  should 
be  an  understanding  between  the  users  and  the 
developers  on  Key  Performance  Parameters 


(KPPs)  in  order  to  identify  the  requirements  that 
are  most  important  and  critical  to  program  suc¬ 
cess.  The  Design  Reference  Mission  Profile  and 
Design  Requirements  templates  in  DoD  4245.7- 
M  and  the  Program  Documentation  Evaluation 
Risk  Identification  technique  may  be  useful  in 
support  of  this  technique. 

Requirements  should  be  thoroughly  reviewed  to 
identify  those  that  drive  performance.  This  will 
require  the  “flow  down”  of  performance  require¬ 
ments  to  components  and  subassemblies  and  the 
identification  of  technologies/techniques  to  be 
used  in  these  components/subassemblies  that 
may  significantly  affect  the  system’s  ability  to 
meet  users’  needs. 

Designers  should  determine  the  sensitivity  of 
system  performance  to  the  requirements  and 
threat  and  identify  risk  drivers.  Models  and  simu¬ 
lations  are  useful  tools  to  determine  this  sensi¬ 
tivity.  For  example,  the  U.S.  Army  Materiel  Sys¬ 
tem  Analysis  Activity  (AMSAA)  has  such  an 
analytic  model,  the  AMSAA  Risk  Assessment 
Methodology. 
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The  PMWS  can  also  be  useful.  The  risk  identi¬ 
fied  in  this  technique  should  be  documented  in  a 
program- specific  format,  such  as  a  RIF  (see 
Annex  B). 

5.4.5  Cost  Risk  Assessment 

5.4.5. 1  Description.  This  technique  provides 
a  program-level  cost  estimate  at  completion 
(EAC)  that  is  a  function  of  performance  (tech¬ 
nical),  and  schedule  risks.  It  uses  the  results  of 
previous  assessments  of  WBS  elements  and  cost 
probability  distributions  developed  for  each  of 
the  elements.  These  individual  WBS  elements 
are  aggregated  using  a  Monte  Carlo  simulation 
to  obtain  a  probability  distribution  of  the  pro¬ 
gram-level  cost  EAC  probability  distribution 
function.  These  results  are  then  analyzed  to 
determine  the  actual  risk  of  cost  overruns  and 
to  identify  the  cost  drivers. 

The  use  of  these  cost  probability  distributions  as 
the  basis  for  the  program- level  cost  estimate  re¬ 
sults  in  a  more  realistic  EAC  than  the  commonly 
used  single  point  estimates  for  WBS  elements, 
since  they  address  both  the  probability  of  occur¬ 
rence  and  consequences/impacts  of  potential  risk 
events.  Their  use  also  eliminates  a  major  cause 
of  underestimating  (use  of  point  estimates)  and 
permits  the  evaluation  of  performance  (techni¬ 
cal)  or  schedule  causes  of  cost  risk.  Thus,  this 
technique  provides  a  basis  for  the  determination 
of  an  “acceptable”  level  of  cost  risk. 

This  technique  can  be  used  in  any  of  the  acqui¬ 
sition  phases,  preferably  at  least  once  per  phase 
beginning  in  the  CR  Phase  although  suitable 
data  or  organization  may  not  exist  until  the  TD 
Phase  or  System  Integration  (SI)  Part  of  the 
SDD  Phase  in  some  cases.  It  should  be  used  in 
conjunction  with  performance  (technical)  and 
schedule  risk  assessments  and  may  be  performed 
by  small  Government-industry  teams  consist¬ 
ing  of  risk  analysts,  cost  analysts,  schedule  ana¬ 
lysts  and  technical  experts  who  understand  the 


significance  of  previous  performance  and 
schedule  risk  assessments.  They  should  report 
to  the  Program  IPT.  This  technique  requires 
close  and  continuous  cooperation  among  cost 
analysts  and  knowledgeable  technical  person¬ 
nel  and  the  support  of  the  prime  contractor’s 
senior  management  to  help  get  valid  cost  data. 

5.4.5.2  Procedures.  Figure  5-8  depicts  the  pro¬ 
cess  used  in  applying  this  technique.  The  first 
step  is  to  identify  the  lowest  WBS  level  for  which 
cost  probability  distribution  will  be  constructed. 
The  level  selected  will  depend  on  the  program 
phase;  e.g.,  during  the  CR  Phase,  it  may  not  be 
possible  to  go  beyond  level  2  or  3,  simply  be¬ 
cause  the  WBS  has  not  yet  been  developed  to 
lower  levels.  As  the  program  advances  into  sub¬ 
sequent  phases  and  the  WBS  is  expanded,  it  will 
be  possible  and  necessary  to  go  to  lower  levels 
(4,  5,  or  lower).  Specific  performance  (techni¬ 
cal)  and  schedule  risks  are  then  identified  for  these 
WBS  elements. 

To  develop  the  WBS  elements  cost  probability 
distributions,  the  team,  working  with  the  prime 
contractor’s  WBS  element  managers,  determines 
the  cost  range  for  each  element  being  investi¬ 
gated.  The  cost  range  encompasses  cost  estimat¬ 
ing  uncertainty,  schedule  risk,  and  technical  risk. 
The  validity  of  the  cost  data  used  to  construct 
the  distribution  is  critical.  In  fact,  collecting  good 
data  is  the  largest  part  of  the  cost  risk  job.  Con¬ 
sequently,  PMOs  should  place  major  emphasis 
on  this  effort. 

The  element  cost  probability  distributions  are 
aggregated  and  evaluated  using  a  Monte  Carlo 
simulation  program.  All  Monte  Carlo  processes 
contain  limitations,  but  they  are  more  informa¬ 
tive  than  point  estimates.  Any  number  of  these 
simulations  are  readily  available  to  perform  this 
aggregation,  and  one  that  meets  the  specific 
needs  of  the  program  should  be  selected.  The 
results  of  this  step  will  be  a  program-level  cost 
EAC  and  a  cost  distribution  that  shows  the 
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cumulative  probability  associated  with  differ¬ 
ent  cost  values.  These  outputs  are  then  analyzed 
to  determine  the  level  of  cost  risk  and  to  iden¬ 
tify  the  specific  cost  drivers.  Cost  risk  is  deter¬ 
mined  by  comparing  the  EAC  with  the  cost 
baseline  developed  as  part  of  the  acquisition 
program  baseline.  Since  the  EAC  and  program 
cost  distribution  are  developed  from  WBS  ele¬ 
ment  risk  assessments,  it  is  possible  to  deter¬ 
mine  the  cost  risk  drivers.  The  cost  drivers  can 
also  be  related  back  to  the  appropriate  perfor¬ 
mance  and  schedule  risks.  The  results  of  the 
analysis  (cost  risks  and  drivers)  should  be  docu¬ 
mented  in  RIFs. 

5.4.6  Quantified  Schedule  Risk 
Assessment 

5.4.6. 1  Description.  This  technique  provides  a 
means  to  determine  program-level  schedule  risk 
as  a  function  of  risk  associated  with  various  ac¬ 
tivities  that  compose  the  program.  It  estimates 
the  program-level  schedule  by  developing  prob¬ 
ability  distributions  for  each  activity  duration  and 
aggregating  these  distributions  using  a  Monte 
Carlo  simulation  or  other  analytical  tools.  The 
resulting  program-level  schedule  is  then  analyzed 


to  determine  the  actual  schedule  risk  and  to  iden¬ 
tify  the  schedule  drivers. 

This  technique  expands  the  commonly  used 
Critical  Path  Method  (CPM)  of  developing  a 
program  schedule  to  obtain  a  realistic  estimate 
of  schedule  risk.  The  basic  CPM  approach  uses 
single  point  estimates  for  the  duration  of  pro¬ 
gram  activities  to  develop  the  program’s  expected 
duration  and  schedule.  It  invariably  leads  to  un¬ 
derestimating  the  time  required  to  complete  the 
program  and  schedule  overruns,  primarily  be¬ 
cause  the  point  estimates  do  not  adequately  ad¬ 
dress  the  uncertainty  inherent  in  individual  ac¬ 
tivities.  The  uncertainty  can  be  caused  by  a  num¬ 
ber  of  factors  and  may  be  a  reflection  of  the  risk 
present  in  the  activity. 

The  quantified  schedule  technique  accounts  for 
uncertainty  by  using  a  range  of  time  that  it  will 
take  to  complete  each  activity  instead  of  single 
point  estimates.  These  ranges  are  then  combined 
to  determine  the  program-level  schedule  esti¬ 
mate.  This  approach  enables  PMs  to  estimate 
early  in  a  program  if  there  is  a  significant  prob¬ 
ability/likelihood  of  overrunning  the  program 
schedule  and  by  how  much.  It  also  identifies  high 
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risk  program  activities  that  may  or  may  not  be 
on  the  program  “critical  path.” 

This  technique  can  be  used  in  any  acquisition 
phase  beginning  with  the  completion  of  the  first 
statement  of  work.  The  schedule  probability  dis¬ 
tribution  function  for  each  key  activity  should 
be  developed  as  soon  as  the  activity  is  included 
in  the  master  schedule.  The  distribution  func¬ 
tions  should  be  periodically  reviewed  and  re¬ 
vised,  if  necessary,  at  least  once  per  phase.  The 
technique  should  be  applied  by  a  small  Gov¬ 
ernment-industry  team  consisting  of  schedule 
analysts  and  technical  experts  who  understand 
the  significance  of  prior  risk  performance 
assessments. 

5.4.6.2  Procedures.  Figure  5-9  shows  the  pro¬ 
cess  used  in  this  technique.  The  first  step  is  to 
identify  the  lowest  activity  level  for  which  dura¬ 
tion/schedule  probability  distribution  functions  will 
be  constructed.  The  WBS  should  be  used  as  the 
starting  point  for  identifying  activities  and  con¬ 
structing  a  network  of  activities.  The  WBS  level 
selected  will  depend  on  the  program  phase. 

Next,  the  contractor  should  construct  a  CPM 
schedule  for  these  activities.  To  develop  the 


activity  duration  probability  distribution  func¬ 
tions,  the  team,  working  with  the  prime 
contractor’s  WBS  element  managers,  determines 
and  analyzes  duration  range  for  each  activity 
being  investigated.  This  analysis  should  be 
done  by  schedule  analysts  working  closely  with 
knowledgeable  technical  people. 

The  activity  duration  probability  distributions  are 
aggregated  using  a  Monte  Carlo  simulation  pro¬ 
gram,  such  as  ©Risk,  Risk-i-  for  Microsoft 
Project,  or  Crystal  Ball.  The  result  of  this  step 
is  a  program-level  schedule  and  distribution 
function  that  shows  the  cumulative  probability 
associated  with  different  duration  values.  These 
outputs  are  then  analyzed  to  determine  the  level 
of  schedule  risk  and  to  identify  the  specific 
schedule  drivers.  Risk  is  determined  by  com¬ 
paring  the  program- level  schedule  with  the  de¬ 
terministic  schedule  baseline  developed  as  part 
of  the  acquisition  program  baseline.  The  fact 
that  the  schedule  and  distribution  are  developed 
from  WBS  element  risk  assessments  makes  it 
possible  to  determine  the  schedule  risk  drivers. 
These  drivers  can  also  be  related  back  to  the  ap¬ 
propriate  performance  risks.  The  results  of  the 
analysis  (schedule  risks  and  drivers)  should  be 
documented  in  RIFs.  The  analysis  requires 
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continued  close  cooperation  between  the  sched¬ 
ule  analysts  and  technical  personnel  familiar  with 
the  details  of  the  program. 

5.4.7  Expert  Interviews 

5.4.7. 1  Description.  A  difficult  part  of  the  risk 
management  process  is  data  gathering.  This  tech¬ 
nique  provides  a  means  for  collecting  risk-re¬ 
lated  data  from  subject-matter  experts  and  from 
people  who  are  intimately  involved  with  the  vari¬ 
ous  aspects  of  the  program.  It  relies  on  “expert” 
judgment  to  identify  and  analyze  risk  events, 
develop  alternatives,  and  provide  “analyzed” 
data.  It  is  used  almost  exclusively  in  a  support 
role  to  help  develop  technical  data,  such  as  prob¬ 
ability  and  consequences/impacts  information, 
required  by  a  primary  risk  assessment  technique. 
It  can  address  all  the  functional  areas  that  make 
up  the  critical  risk  areas  and  processes,  and  can 
be  used  in  support  of  risk  handling. 

Expert  judgment  is  a  sound  and  practical  way 
of  obtaining  necessary  information  that  is  not 
available  elsewhere  or  practical  to  develop  us¬ 
ing  engineering  or  scientific  techniques.  How¬ 
ever,  interviewers  should  be  aware  that  expert 
opinions  may  be  biased  because  of  over-reliance 


on  certain  information  and  neglect  of  other  in¬ 
formation;  unwarranted  confidence;  the  tendency 
to  recall  most  frequent  and  most  recent  events;  a 
tendency  to  neglect  rare  events;  and  motivation. 
Results  may  have  to  be  tempered  because  of 
these  biases. 

5.4.7.2  Procedures.  Figure  5-10  depicts  the  pro¬ 
cess  used  in  this  technique.  The  first  step  in  the 
process  is  to  identify  risk  areas  and  processes  that 
are  to  be  evaluated  using  the  expert  interview  tech¬ 
nique.  Other  techniques  described  in  this  section 
(e.g.,  WBS  Risk  Assessment,  Process  Risk  As¬ 
sessment,  etc.)  can  be  used  for  this  purpose. 

Once  the  areas  and  processes  are  known,  subject- 
matter  experts  and  program/contractor  person¬ 
nel  knowledgeable  of  the  areas  and  processes 
should  be  identified  to  be  interviewed.  Similarly, 
qualified  interviewers  should  be  selected  for  each 
area  and  process. 

Interviewers  should  prepare  themselves  by  pre¬ 
paring  a  strategy  and  selecting  a  methodology 
for  analysis  and  quantification  of  data.  The  ref¬ 
erences  list  sources  for  practical  techniques  for 
quantifying  expert  judgment.  (See  Appendix  D 
for  additional  guidance  in  this  area.) 
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After  the  interview,  evaluators  analyze  the  data 
for  consistency,  resolve  any  issues,  and  docu¬ 
ment  the  results.  Commercial  “Groupware”  soft¬ 
ware  is  available  to  assist  in  compiling  and  docu¬ 
menting  the  results  of  interviews. 

5.4.8  Analogy  Comparison/ 

Lessons-Learned  Studies 

5.4.8.1  Description.  This  technique  uses  les¬ 
sons  learned  and  historical  information  about 
the  risk  associated  with  programs  that  are  simi¬ 
lar  to  the  new  system  to  identify  the  risk  asso¬ 
ciated  with  a  new  program.  It  is  normally  used 
to  support  other  primary  risk  assessment  tech¬ 
niques,  e.g.,  Product  (WBS)  Risk  Assessment, 
Process  Risk  Assessment,  etc.  The  technique  is 
based  upon  the  concept  that  “new”  programs  are 
originated  or  evolved  from  existing  programs  or 
simply  represent  a  new  combination  of  existing 
components  or  subsystems.  This  technique  is 
most  appropriate  when  systems  engineering  and 
systems  integration  issues,  plus  software 


development,  are  minimal.  A  logical  extension 
of  this  premise  is  that  key  insights  can  be  gained 
concerning  aspects  of  a  current  program’s  risks 
by  examining  the  successes,  failures,  problems, 
and  solutions  of  similar  existing  or  past  programs. 
This  technique  addresses  all  the  functional  areas 
that  make  up  the  critical  risk  areas  and  processes. 

5.4.8.2  Procedures.  Figure  5-11  depicts  the 
process  used  in  this  technique.  The  first  step  in 
this  approach  is  to  select  or  develop  a  baseline 
comparison  system  (BCS)  that  closely  approxi¬ 
mates  the  characteristics  of  the  new  system/ 
equipment  to  as  low  a  level  as  possible  and  uses 
the  processes  similar  to  those  that  are  needed  to 
develop  the  new  system.  For  processes,  indus¬ 
try-wide  best  practices  should  be  used  as  a 
baseline.  The  PMWS  is  a  useful  tool  for  identi¬ 
fying  these  best  practices. 

Relevant  BCS  data  are  then  collected,  analyzed, 
and  compared  with  the  new  system  require¬ 
ments.  The  BCS  data  may  require  adjustment  to 
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Figure  5-1 1 .  Analogy  Comparison/Lessons-Learned  Studies  Top-Level  Diagram 
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make  a  valid  comparison;  for  example,  apply 
appropriate  inflation  indices  for  cost  compari¬ 
sons,  adjust  design  schedule  for  software  evolu¬ 
tion  versus  software  development,  etc.  The  com¬ 
parisons  can  be  a  major  source  of  risk  assess¬ 
ment  data  and  provide  some  indication  of  areas 
that  should  be  investigated  further.  This  tech¬ 
nique  is  especially  useful  as  a  front-end  analysis 
of  a  new  start  program. 

5.5  RISK  PRIORITIZATION 

5.5.1  Description 

This  technique  provides  a  means  to  prioritize  the 
risks  present  in  a  program.  It  is  a  part  of  risk  analy¬ 
sis.  The  prioritized  list  provides  the  basis  for  de¬ 
veloping  handling  plans,  preparing  a  handling  task 
sequence  list,  and  allocating  handling  resources. 

When  using  this  technique,  PMs  establish 
definitive  criteria  to  evaluate  the  risks,  such  as, 
probability  (probability/likelihood)  of  failure, 
(Pp),  and  consequence/impact  of  failure  (Cp), 
along  with  any  other  factors  considered  ap¬ 
propriate.  The  risks  are  evaluated  using  qualita¬ 
tive  expert  judgment  and  multi- voting  methods 
to  prioritize  and  aggregate  risks.  (See  Refer- 
ences-SEI,  Continuous  Risk  Management,  1996, 
for  a  discussion  of  multi-voting  methods.)  A 


qualitative  approach  using  subject-matter  experts 
is  generally  preferred  in  this  technique  because 
of  the  tendency  to  rely  on  ordinal  values  to 
describe  P„,  Cc  and  the  inherent  inaccuracies 
resulting  from  any  attempts  to  use  quantifiable 
methods  derived  from  raw  (uncalibrated)  ordinal 
scales. 

This  technique  should  be  used  appropriately 
during  the  CR  and  TD  Phases,  and  the  SI  and 
SD  parts  of  the  SDD  Phase;  at  the  conclusion  of 
a  major  risk  assessment  undertaking;  when  there 
has  been  a  significant  change  in  the  acquisition 
strategy;  when  risk  monitoring  indicates  signifi¬ 
cant  changes  in  the  status  of  a  number  of  risks, 
and  prior  to  a  milestone  review. 

The  PMO  risk  management  coordinator  (if 
assigned)  may  function  as  a  facilitator  and 
support  the  program  IPT  in  applying  this 
technique. 

5.5.2  Procedures 

Figure  5-12  depicts  the  process  used  to  prioritize 
the  risks  present  in  a  program.  The  inputs  of 
this  process  are  risks  that  have  been  identified. 

The  evaluation  team,  through  consensus  or  as 
directed  by  the  Risk  Management  Plan,  selects 
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the  prioritization  criteria.  PF  and  CF  should  always 
be  part  of  the  criteria,  along  with  any  other  ap¬ 
propriate  factors.  Urgency,  an  indication  of  the 
time  available  before  the  procedures  for  handling 
the  specific  risk  must  be  initiated,  is  often  con¬ 
sidered  in  the  evaluation.  The  PM  may  also 
choose  to  rank-order  the  prioritization  criteria, 
e.g.,  consequence/impact  is  more  important  than 
probability. 

A  multi-voting  method  is  useful  to  prioritize  risks 
(see  References-Scholtes,  1988;  Linstone,  1975). 
The  Delphi  method  is  a  simple  and  effective 
method  of  arriving  at  a  consensus  among  a  group 
of  experts.  The  procedure  is  for  team  members  to 
vote  on  the  priority  of  each  risk  and  tally  the  re¬ 
sults,  which  are  fed  back  to  the  team.  Team  mem¬ 
bers  vote  again  and  the  process  is  repeated  until 
no  changes  occur  in  the  results.  It  is  normal  to 
reach  the  final  outcome  within  a  few  voting  ses¬ 
sions.  If  there  are  a  large  number  of  risks,  they 
may  be  broken  into  smaller  groups  for  ranking. 
As  a  general  rule,  no  more  than  10  items  should 
be  prioritized  per  vote.  The  results  of  the  series  of 
votes  are  documented  in  the  prioritized  list  of  risks. 


specify  prioritization  criteria  and  prescribe  the 
format  of  the  prioritized  list  of  risks. 

5.5.2.1  Risk  Aggregation.  Figure  5-13  shows 
the  process  for  this  technique,  which  relies  on 
qualitative  judgment  and  multi- voting  methods 
to  summarize  risks  at  the  critical  risk  area  and 
process  level  in  terms  of  Pp  and  C  .  The  risks 
identified  in  the  RIFs  and  the  prioritized  list  of 
risks  are  first  grouped  according  to  critical  risk 
areas  and  processes,  and  listed  in  priority 
sequence. 

Within  each  area  and  process,  the  individual  risks 
are  evaluated  against  a  set  of  established  criteria 
to  determine  the  overall  aggregate  risk  rating  for 
the  area/process.  Aggregation  criteria  needs  to  be 
established  separately  for  Pp  and  Cp  ;  Pp  and  Cp 
should  not  be  combined  into  a  single  index,  e.g., 
moderate  risk.  Examples  of  aggregation  criteria 
include:  (1)  most  undesirable  Ppand  CF  of  all  the 
risks  within  a  risk  area  or  process  becomes  the 
aggregated  values  for  the  area  or  process,  or  (2) 
the  Pp  and  CF  for  each  area  or  process  represents 
the  mean  value  for  that  area  or  process. 


PM  guidance,  which  operates  as  a  technique  The  team  then  votes  on  each  risk  area  and  pro¬ 
control  function,  can  be  used,  for  example,  to  cess  to  determine  its  rating  for  Pp  and  CF,  and 
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the  results  are  documented.  In  addition  to  the 
Ppand  Cp  ratings  for  each  critical  risk  area  and 
process,  those  risks  that  tend  to  “drive”  the  ag¬ 
gregate  risk  rating  for  the  area/process  should 
be  included  in  a  list  of  aggregated  risks  to  give 
substance  to  the  aggregated  ratings,  e.g.,  all 
risks  in  which  either  Ppor  CF  are  rated  as  high. 
Figure  5-14  provides  a  sample  list  of  aggregated 
risks. 

Risk  Matrix  is  a  software  tool  that  is  designed  to 
aid  in  managing  the  identification,  rating,  and 
prioritization  of  key  risks  that  might  affect  a 
project.  It  provides  a  structured  method  for 
prioritizing  project  risks  and  for  tracking  the  sta¬ 
tus  and  effects  of  risk-handling  efforts.  The  ma¬ 
jor  feature  that  Risk  Matrix  offers  the  program 
office  is  a  means  to  both  rate  and  rank  program 
risks.  This  is  helpful  in  differentiating  among 
risks  that  have  the  same  rating.  For  example,  if  a 
program  has  eight  risks  that  the  program  office 
has  evaluated/rated  as  high,  Risk  Matrix  provides 
the  means  to  rank  them  in  order  of  severity.  The 
user  can  use  this  ranking  as  a  guide  to  help  fo¬ 
cus  risk-handling  efforts.  Risk  Matrix  was  de¬ 
veloped  by  the  Air  Force  Electronic  Systems 
Center  (ESC)  and  The  Mitre  Corporation  and  is 
available  to  program  offices  at  no  cost.  Another 
useful  software  tool  to  use  in  voting  on  risks  is 
“Expert  Choice” — based  on  the  Analytical  Hi¬ 
erarchy  Process  (AHP).  Whatever  software  tool 
is  used,  the  analyst  should  recognize  that  a 


number  of  inherent  limitation  exist  with  such  soft¬ 
ware  tools,  (e.g.,  unintentionally  biasing  the  vot¬ 
ing  process)  that  can  lead  to  erroneous  results. 

5.6  RISK-HANDLING  TECHNIQUES 

5.6.1  General  (e.g.,  Moderate  and 
High  Risk-Rated  Items) 

After  the  program’s  risks  have  been  assessed, 
the  PM  must  develop  approaches  to  handle 
significant  ones  by  analyzing  various  handling 
techniques  and  selecting  those  best  fitted  to  the 
program’s  circumstances.  The  PM  should  reflect 
these  approaches  in  the  program’s  acquisition 
strategy  and  include  the  specifics  on  what  is  to 
be  done  to  deal  with  the  risk,  when  it  should  be 
accomplished,  who  is  responsible,  and  the  cost 
and  schedule  impact. 

As  described  in  Chapter  2,  there  are  essentially 
four  risk-handling  techniques,  or  options.  Risk 
avoidance  eliminates  the  sources  of  high  risk  and 
replaces  them  with  a  lower-risk  solution.  Risk 
transfer  is  the  reallocation  of  risk  from  one  part 
of  the  system  to  another,  or  the  reallocation  of  risks 
between  the  Government  and  the  prime  contrac¬ 
tor  or  within  Government  agencies.  Risk  control 
manages  the  risk  in  a  manner  that  reduces  the  prob¬ 
ability/likelihood  of  its  occurrence  and/or  mini¬ 
mizes  and  mitigates  the  risk’s  effect  on  the  pro¬ 
gram.  Risk  assumption  is  the  acknowledgment 
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of  the  existence  of  a  particular  risk  situation  and 
a  conscious  decision  to  accept  the  associated  level 
of  risk  without  engaging  in  any  special  efforts  to 
control  it.  There  is  a  tendency  on  many  programs 
to  select  “control”  as  the  risk-handling  option 
without  seriously  evaluating  assumption,  avoid¬ 
ance,  and  transfer.  This  is  unwise,  since  control 
may  not  be  the  best  option,  or  even  appropriate 
option  in  some  cases.  An  unbiased  assessment 
of  risk-handling  options  should  be  performed  to 
determine  the  most  appropriate  option. 

In  determining  the  “best”  overall  risk-handling 
strategy  to  be  adopted,  a  structured  approach 
should  be  taken. 

A  structured  approach  for  developing  a  risk¬ 
handling  strategy  has  been  described  by  Dr. 
Edmund  Conrow  in  his  book  Effective  Risk 
Management:  Some  Keys  to  Success.  (See  Ref¬ 
erence.)  A  risk-handling  strategy  is  composed 
of  the  selected  risk-handling  option  and  the  spe¬ 
cific  implementation  activity.  The  risk-handling 
option  is  first  chosen,  then  the  best  implemen¬ 
tation  activity  is  picked  for  the  selected  option. 
This  avoids  a  common  mistake  — choosing  the 
implementation  activity  without  first  evaluat¬ 
ing  all  four  risk-handling  (generic)  options.  In 
cases  where  a  relatively  high  risk  exists,  or 
where  the  other  circumstances  dictate,  one  or 
more  backup  risk-handling  strategies  may  be 
needed.  In  these  cases,  the  selection  process  is 
used  again  to  choose  the  option  and  implemen¬ 
tation  activity.  The  backup  strategy  may  have  a 
different  option  than  used  in  the  primary  risk¬ 
handling  strategy,  and  will  certainly  have  a  dif¬ 
ferent  implementation  activity. 

For  each  evaluated  event  risk,  all  potentially 
applicable  options  or  techniques  should  be 
identified  and  evaluated,  using  the  following 
criteria: 

•  Feasibility  -  Feasibility  is  the  ability  to  im¬ 
plement  the  handling  technique/option  and 


includes  an  evaluation  of  the  potential  impact 
of  the  technique/option  in  the  following  areas: 

-  Technical  considerations,  such  as  testing, 
manufacturing,  and  maintainability, 
caused  by  design  changes  resulting  from 
risk-handling  techniques. 

-  Adequacy  of  budget  and  schedule 
flexibility  to  apply  the  technique. 

-  Operational  issues  such  as  usability  (man- 
machine  interfaces),  transportability,  and 
mobility. 

-  Organizational  and  resource  considerations, 
e.g.,  manpower,  training,  and  structure. 

-  Environmental  issues,  such  as  the  use  of 
hazardous  materials  to  reduce  technical 
risk. 

-  External  considerations  beyond  the  imme¬ 
diate  scope  of  the  program,  such  as  the 
impact  on  other  complementary  systems 
or  organizations. 

•  Cost  and  schedule  implications  -  The  risk¬ 
handling  techniques  have  a  broad  range  of 
cost  implications  in  terms  of  dollars,  as  well 
as  other  limited  resources,  e.g.,  critical  ma¬ 
terials  and  national  test  facilities.  The  mag¬ 
nitude  of  the  cost  and  schedule  implications 
will  depend  on  circumstances  and  can  be  as¬ 
sessed  using  such  techniques  as  cost-benefit 
analyses  and  the  cost  and  schedule  assess¬ 
ment  techniques  previously  described.  The 
approval  and  funding  of  risk-handling  tech¬ 
niques  should  be  part  of  the  trade-off  pro¬ 
cess  that  establishes  and  refines  the  CAIV 
cost  and  performance  goals. 

•  Effect  on  the  system’s  technical  perfor¬ 
mance  -  The  risk-handling  techniques  may 
affect  the  system’s  capability  to  achieve  the 
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required  technical  performance  objectives. 
This  impact  must  be  clearly  understood  be¬ 
fore  adopting  a  specific  technique.  As  the  risk¬ 
handling  techniques  are  assessed,  the  PMO 
should  attempt  to  identify  any  additional  pa¬ 
rameters  that  may  become  critical  to  techni¬ 
cal  performance  as  a  result  of  implementing 
them.  Trade  studies  and  sensitivity  analyses 
can  be  useful  in  determining  the  expected  ef¬ 
fectiveness  of  this  approach. 

Once  the  risk-handling  technique  is  selected,  a 
set  of  program  management  indicators  should 
be  developed  to  provide  feedback  on  program 
progress,  effectiveness  of  the  risk-handling 
options  selected,  and  information  necessary  to 
manage  the  program.  These  indicators  should 
consist  of  cost  and  scheduling  data,  technical  per¬ 
formance  measures,  and  program  metrics. 

Subsequent  paragraphs  in  this  section  describe 
the  various  risk-handling  technique:  Risk  Con¬ 
trol,  Avoidance,  Assumption,  Transfer  (CAAT). 

5.6.2  Risk  Control 

5.6.2. 1  Description.  In  this  risk-handling 
technique,  the  Government  and  contractor  take 
active  steps  to  reduce  the  probability /likelihood 
of  a  risk  event  occurring  and  to  reduce  the 
potential  impact  on  the  program.  The  common 
name  for  the  control  option  is  “mitigation.”  Most 
risk-control  steps  share  two  features:  they  require 
a  commitment  of  program  resources,  and  they 
may  require  additional  time  to  accomplish  them. 
Thus,  the  selection  of  risk-control  actions  will 
undoubtedly  require  some  tradeoff  between 
resources  and  the  expected  benefit  of  the  actions. 
Some  of  the  many  risk-control  actions  include 
the  following: 

Multiple  Development  Efforts  -  The  use  of 

two  or  more  independent  design  teams  (usually 
two  separate  contractors,  although  it  could  also 
be  done  internally)  to  create  competing  systems 


in  parallel  that  meet  the  same  performance 
requirements. 

Alternative  Design  -  Sometimes,  a  design  option 
may  include  several  risky  approaches,  of  which 
one  or  more  must  come  to  fruition  to  meet  system 
requirements.  However,  if  the  PMO  studies  the 
risky  approaches,  it  may  be  possible  to  discover  a 
lower-risk  approach  (with  a  lower  performance 
capability).  These  lower-risk  approaches  could  be 
used  as  backups  for  those  cases  where  the  pri¬ 
mary  approach(es)  fail  to  mature  in  time.  This  op¬ 
tion  presumes  there  is  some  trading  room  among 
requirements.  Close  coordination  between  the 
developer  and  the  user  is  necessary  to  implement 
lower  capability  options. 

Trade  Studies  -  Systems  engineering  decision 
analysis  methods  include  trade  studies  to  solve 
a  complex  design  problem.  The  purpose  of  the 
trade  studies  is  to  integrate  and  balance  all 
engineering  requirements  in  the  design  of  a 
system.  A  properly  done  trade  study  considers 
risks  associated  with  alternatives. 

Early  Prototyping  -  The  nature  of  a  risk  can  be 
evaluated  by  a  prototype  of  a  system  (or  its  criti¬ 
cal  elements)  built  and  tested  early  in  the  system 
development.  The  results  of  the  prototype  can  be 
factored  into  the  design  and  manufacturing  pro¬ 
cess  requirements.  In  addition  to  full-up  systems, 
prototyping  is  very  useful  in  software  develop¬ 
ment  and  in  determining  a  system’s  man-machine 
interface  needs.  The  key  to  making  prototyping 
successful  as  a  risk-control  tool  is  to  minimize  the 
addition  of  new  requirements  to  the  system  after 
the  prototype  has  been  tested  (i.e.,  requirement 
changes  not  derived  from  experience  with  the 
prototype).  Also,  the  temptation  to  use  the  proto¬ 
type  design  and  software  without  doing  the  nec¬ 
essary  follow-on  design  and  coding/manufactur¬ 
ing  analyses  should  be  avoided. 

Incremental  Development  -  Incremental 
development  is  completion  of  the  system 
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design  and  deployment  in  steps,  relying  on 
pre-planned  product  improvements  (P3I)  or 
software  improvements  after  the  system  is  de¬ 
ployed  to  achieve  the  final  system  capability. 
Usually,  these  added  capabilities  are  not  in¬ 
cluded  originally  because  of  the  high  risk  that 
they  will  not  be  ready  along  with  the  remain¬ 
der  of  the  system.  Hence,  development  is  split, 
with  the  high-risk  portion  given  more  time  to 
mature.  The  basic  system,  however,  incorpo¬ 
rates  the  provisions  necessary  to  include  the 
add-on  capabilities.  Incremental  development 
of  the  initial  system  requirements  are  achieved 
by  the  basic  system. 

Technology  Maturation  Efforts  -  Technology 
maturation  is  an  off-line  development  effort  to 
bring  an  element  of  technology  to  the  neces¬ 
sary  level  so  that  it  can  be  successfully  incor¬ 
porated  into  the  system  (usually  done  as  part  of 
the  technology  transition  process).  Normally, 
technology  maturation  is  used  when  the  desired 
technology  will  replace  an  existing  technology, 
which  is  available  for  use  in  the  system.  In  those 
cases,  technology  maturation  efforts  are  used 
in  conjunction  with  P3I  efforts.  However,  it  can 
also  be  used  when  a  critical,  but  immature,  tech¬ 
nology  is  needed.  In  addition  to  dedicated 
efforts  conducted  by  the  PMO,  Service  or  DoD- 
wide  technology  improvement  programs  and 
advanced  technology  demonstrations  by 
Government  laboratories  as  well  as  industry 
should  be  considered. 

Robust  Design  -  This  approach  uses  advanced 
design  and  manufacturing  techniques  that  pro¬ 
mote  achieving  quality  through  design.  It  nor¬ 
mally  results  in  products  with  little  sensitivity  to 
variations  in  the  manufacturing  process. 

Reviews,  Walk  Throughs,  and  Inspections  - 

These  three  risk  control  actions  can  be  used  to 
reduce  the  probability /likelihood  and  potential 
consequences/impacts  of  risks  through  timely  as¬ 
sessments  of  actual  or  planned  events  in  the 


development  of  the  product.  They  vary  in  the 
degree  of  formality,  level  of  participants,  and 
timing. 

Reviews  are  formal  sessions  held  to  assess  the 
status  of  the  program,  the  adequacy  and  suffi¬ 
ciency  of  completed  events,  and  the  intentions 
and  consistency  of  future  events.  Reviews  are 
usually  held  at  the  completion  of  a  program 
phase,  when  significant  products  are  available. 
The  team  conducting  the  review  should  have  a 
set  of  objectives  and  specific  issues  to  be 
addressed.  The  results  should  be  documented 
in  the  form  of  action  items  to  be  implemented 
by  the  PMO  or  contractor.  The  type  of  review 
will  dictate  the  composition  of  the  review  team, 
which  may  include  developers,  users,  managers, 
and  outside  experts. 

A  walk  through  is  a  technique  that  can  be  very 
useful  in  assessing  the  progress  in  the  develop¬ 
ment  of  high-  or  moderate-risk  components, 
especially  software  modules.  It  is  less  formal 
than  a  review,  but  no  less  rigorous.  The  person 
responsible  for  the  development  of  the  compo¬ 
nent  “walks  through,”  the  product  development 
(to  include  perceptions  of  what  is  to  be  done, 
how  it  will  be  accomplished,  and  the  schedule) 
with  a  team  of  subject-matter  experts.  The  team 
reviews  and  evaluates  the  progress  and  plans 
for  developing  the  product  and  provides  imme¬ 
diate  and  less  formal  feedback  to  the  responsible 
person,  thus  enabling  improvements  or  correc¬ 
tive  actions  to  be  made  while  the  product  is  still 
under  development.  This  technique  is  applied 
during  the  development  phases,  as  opposed  to 
reviews,  which  are  normally  held  at  the  comple¬ 
tion  of  a  phase  or  product. 

Inspections  are  conducted  to  evaluate  the  cor¬ 
rectness  of  the  product  under  development  in 
terms  of  its  design,  implementation,  test  plans, 
and  test  results.  They  are  more  formal  and  rig¬ 
orous  than  either  reviews  or  walk  throughs  and 
are  conducted  by  a  team  of  experts  following  a 
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very  focused  set  of  questions  concerning  all 
aspects  of  the  product. 

Design  of  Experiments  -  This  is  an  engineer¬ 
ing  tool  that  identifies  critical  design  factors  that 
are  difficult  to  meet. 

Open  Systems  -  This  approach  involves  the  use 
of  widely  accepted  commercial  specifications 
and  standards  for  selected  system  interfaces, 
products,  practices,  and  tools.  It  provides  the 
basis  for  reduced  life-cycle  costs,  improved  per¬ 
formance,  and  enhanced  interoperability, 
especially  for  long-life  systems  with  short-life 
technologies.  Properly  selected  and  applied 
commercial  specifications  and  standards  can 
result  in  lower  risk  through  increased  design  flex¬ 
ibility;  reduced  design  time;  more  predictable  per¬ 
formance;  and  easier  product  integration,  sup¬ 
port,  and  upgrade.  However,  a  number  of  chal¬ 
lenges  and  risks  are  associated  with  the  use  of 
the  open  systems  approach  and  must  be  consid¬ 
ered  before  implementation.  These  include  such 
issues  as:  maturity  and  acceptability  of  the  stan¬ 
dard,  and  its  adequacy  for  military  use;  the  loss 
of  control  over  the  development  of  products  used 
in  the  system;  the  amount  of  product  testing  done 
to  ensure  conformance  to  standards;  and  the 
higher  configuration  management  workload  re¬ 
quired. 

See  the  Defense  Acquisition  Deskbook  for  in¬ 
formation  on  the  use  of  open  systems.  (Addi¬ 
tional  information  is  also  available  at  the  Open 
Systems  Joint  Task  Force  Website  at  http:// 
www.acq.osd.mil/osjtf/) 

Use  of  Standard  Items/Software  Module 
Reuse  -  The  use  of  standard  items  and  software 
module  reuse  should  be  emphasized  to  the  ex¬ 
tent  possible  to  minimize  development  risk.  Stan¬ 
dard  items  range  from  components  and  assem¬ 
blies  to  full-up  systems.  A  careful  examination 
of  the  proposed  system  option  will  often  find 
more  opportunities  for  the  use  of  standard  items 


or  existing  software  modules  than  first  consid¬ 
ered.  Even  when  the  system  must  achieve  pre¬ 
viously  unprecedented  requirements,  standard 
items  can  find  uses.  A  strong  program  policy  em¬ 
phasizing  the  use  of  standard  items  and  software 
reuse  is  often  the  key  to  taking  advantage  of  this 
source  of  risk  control.  Standard  items  and  soft¬ 
ware  modules  have  proven  characteristics  that 
can  reduce  risk.  However,  the  PMO  must  be  cau¬ 
tious  when  using  standard  items  in  environ¬ 
ments  and  applications  for  which  they  were  not 
designed.  A  misapplied  standard  item  often  leads 
to  problems  and  failure.  Similarly,  if  the  cycle 
for  a  fielded  product  extends  for  many  years,  it 
is  possible  that  key  software  tools  and  products 
will  become  obsolete  or  will  no  longer  be 
supported.  If  this  occurs,  costly  redesign  may 
result  if  software  re-development  is  necessary. 

Two-Phase  Development  -  This  risk  control 
approach  incorporates  a  formal  risk-reduction 
effort  in  the  initial  part  of  the  SDD  phase.  It 
may  involve  using  two  or  more  contractors 
with  a  down-select  occurring  at  a  predefined 
time  (normally  after  the  preliminary  design  re¬ 
view).  A  logical  extension  of  this  concept  is  the 
“spiral”  development  model,  which  emphasizes 
the  evaluation  of  alternatives  and  risk  assess¬ 
ments  throughout  the  system’s  development  and 
initial  fielding. 

Use  of  Mockups  -  The  use  of  mockups,  espe¬ 
cially  man-machine  interface  mock-ups,  can  be 
used  to  conduct  early  exploration  of  design 
options.  They  can  assist  in  resolving  design 
uncertainties  and  providing  users  with  early 
views  of  the  final  system  configuration. 

Modeling/Simulation  -  The  use  of  modeling 
and  simulation  can  provide  insights  into  a 
system’s  performance  and  effectiveness  sensi¬ 
tivities.  Decision  makers  can  use  performance 
predictions  to  assess  a  system’s  military  worth 
not  only  before  any  physical  prototypes  are 
built,  but  also  throughout  the  system  life  cycle. 
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Modeling  and  simulation  can  help  manage  risk 
by  providing  information  on  design  capabilities 
and  failure  modes  during  the  early  stages  of  de¬ 
sign.  This  allows  initial  design  concepts  to  be 
iterated  without  having  to  build  hardware  for 
testing.  The  T&E  community  can  use  predictive 
simulations  to  focus  the  use  of  valuable  test  as¬ 
sets  on  critical  test  issues.  They  can  also  use  ex¬ 
trapolated  simulations  to  expand  the  scope  of 
evaluation  into  areas  not  readily  testable,  thus 
reducing  the  risk  of  having  the  system  fail  in  the 
outer  edges  of  the  “test  envelope.”  Additionally, 
a  model  can  serve  as  a  framework  to  bridge  the 
missing  pieces  of  a  complete  system  until  those 
pieces  become  available. 

Although  modeling  and  simulation  can  be  a  very 
effective  risk-handling  tool,  it  requires  resources, 
commitment  to  refine  models  as  the  system  un¬ 
der  development  matures,  and  a  concerted  veri¬ 
fication  and  validation  effort  to  ensure  that  deci¬ 
sions  are  based  on  credible  information. 

Key  Parameter  Control  Boards  -  When  a 
particular  parameter  (such  as  system  weight) 
is  crucial  to  achieving  the  overall  program  re¬ 
quirements,  a  control  board  for  that  parameter 
may  be  appropriate.  This  board  has  represen¬ 
tatives  from  all  affected  technical  functions 
and  may  be  chaired  by  the  PM.  It  provides 
management  focus  on  the  parameter  and  sig¬ 
nals  the  importance  of  achieving  the  param¬ 
eter  to  the  technical  community.  If  staffed 
properly  by  all  affected  disciplines,  it  can  also 
help  avoid  sacrificing  other  program  require¬ 
ments  to  achieve  that  requirement. 

Manufacturing  Screening  -  For  programs  in 
late  SDD  and  early  production  and  deployment, 
various  manufacturing  screens  (including  envi¬ 
ronmental  stress  screening  (ESS))  can  be  incor¬ 
porated  into  test  article  production  and  low-rate 
initial  production  to  identify  deficient  manufac¬ 
turing  processes.  ESS  is  a  manufacturing  pro¬ 
cess  for  stimulating  parts  and  workmanship  de¬ 


fects  in  electronic  assemblies  and  units.  These 
data  can  then  be  used  to  develop  the  appropriate 
corrective  actions. 

Test,  Analyze,  and  Fix  (TAAF)  -  TAAF  is  the 
use  of  a  period  of  dedicated  testing  to  identify 
and  correct  deficiencies  in  a  design.  It  was  origi¬ 
nally  conceived  as  an  approach  to  improve 
reliability;  it  can  also  be  used  for  any  system 
parameter  whose  development  could  benefit 
from  a  dedicated  period  of  testing  and  analysis. 
Although  a  valuable  aid  in  the  development 
process,  TAAF  should  not  be  used  in  lieu  of  a 
sound  design  process. 

Demonstration  Events  -  Demonstration  events 
are  points  in  the  program  (usually  tests)  that  are 
used  to  determine  if  risks  are  being  successfully 
abated.  Careful  review  of  the  planned 
development  of  each  risk  area  will  reveal  a  num¬ 
ber  of  opportunities  to  verify  the  effectiveness 
of  the  development  approach.  By  including  a 
sequence  of  demonstration  events  throughout 
the  development,  PMO  and  contractor  person¬ 
nel  can  monitor  the  process  and  identify  when 
additional  efforts  are  needed.  Demonstration 
events  can  also  be  used  as  information-gather¬ 
ing  actions,  as  discussed  before,  and  as  part  of 
the  risk-monitoring  process.  Table  5-2  contains 
examples  of  demonstration  events. 

Process  Proofing  -  When  particular  processes, 
especially  those  of  manufacturing  and  support, 
are  critical  to  achieving  system  requirements,  an 
early  process  proof  demonstration  is  useful  to 
abate  risk.  If  the  initial  proof  is  unsuccessful,  time 
is  still  available  to  identify  and  correct  deficien¬ 
cies  or  to  select  an  alternative  approach. 

No  single  technique  or  tool  is  capable  of  provid¬ 
ing  a  complete  answer — a  combination  must  be 
used.  In  general,  risk-monitoring  techniques  are 
applied  to  follow  through  on  the  planned  actions 
of  the  risk-handling  program.  They  track  and 
evaluate  the  effectiveness  of  handling  activities 
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Item 

Demonstration  Event 

Completion  Date 

Rocket  Motor 

Three  Case  Burst  Tests 

Propellant  Characterization 

Thermal  Barrier  Bond  Tests 

Ignition  and  Safe/Arm  Tests 

Nozzle  Assembly  Tests 

By  completion  of  preliminary  design 

10  Development  Motor  Firings 

—  Temperature  and  Altitude  Cycle 

—  Vibration  and  Shock 

—  Aging 

By  completion  of  final  design 

Central 

Computer 

Test  Breadboard 

Develop/Test  Unique  Microcircuits 

By  completion  of  preliminary  design 

Build/Test  Prototype 

By  completion  of  final  design 

Table  5-2.  Examples  of  Demonstration  Events 


by  comparing  planned  actions  with  what  is  actu¬ 
ally  achieved.  These  comparisons  may  be  as 
straightforward  as  actual  versus  planned  comple¬ 
tion  dates,  or  as  complex  as  detailed  analysis  of 
observed  data  versus  planned  profiles.  In  any  case, 
the  differences  between  planned  and  actual  data 
are  examined  to  determine  status  and  the  need  for 
any  changes  in  the  risk-handling  approach. 

PMO  personnel  should  also  ensure  that  the  in¬ 
dicators/metrics  selected  to  monitor  program 
status  adequately  portray  the  true  state  of  the 
risk  events  and  handling  actions.  Otherwise, 
indicators  of  risks  that  are  about  to  become  prob¬ 
lems  will  go  undetected.  Subsequent  sections 
identify  specific  techniques  and  tools  that  will 
be  useful  to  PMOs  in  monitoring  risks  and  pro¬ 
vide  information  on  selecting  metrics  that  are 
essential  to  the  monitoring  effort.  The  techniques 
focus  primarily  at  the  program  level,  addressing 
cost,  schedule,  and  performance  risks. 

5.6.2.2  Procedures.  Risk  control  involves 
developing  a  risk-reduction  plan,  with  actions 
identified,  resourced,  and  scheduled.  Success  cri¬ 
teria  for  each  of  the  risk-reduction  events  should 
also  be  identified.  The  effectiveness  of  these 


actions  must  be  monitored  using  the  types  of 
techniques  described  in  Section  5.7. 

5.6.3  Risk  Avoidance 

5.6.3. 1  Description.  This  technique  reduces 
risk  through  the  modification  or  elimination  of 
those  operational  requirements,  processes  or 
activities  that  cause  the  risks.  Eliminating  opera¬ 
tional  requirements  requires  close  coordination 
with  the  users.  Since  this  technique  results  in  the 
reduction  of  risk,  it  should  generally  be  initiated 
in  the  development  of  a  risk-handling  plan.  It 
can  be  done  in  parallel  with  the  initial  opera¬ 
tional  requirements  analysis  and  should  be  sup¬ 
ported  by  a  cost-benefit  analysis. 

5.6.3.2  Procedures.  Analyzing  and  reviewing 
the  proposed  system  in  detail  with  the  user  is 
essential  to  determine  the  drivers  for  each  op¬ 
erational  requirement.  Operational  requirements 
scrubbing  involves  eliminating  those  that  have 
no  strong  basis.  This  also  provides  the  PMO  and 
the  user  with  an  understanding  of  what  the  real 
needs  are  and  allows  them  to  establish  accurate 
system  requirements  for  the  critical  performance. 
Operational  requirements  scrubbing  essentially 
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consists  of  developing  answers  to  the  following 
questions: 

•  Why  is  the  requirement  needed? 

•  What  will  the  requirement  provide? 

•  How  will  the  capability  be  used? 

•  Are  the  requirements  specified  in  terms  of 
functions  and  capabilities,  rather  than  a 
specific  design? 

Cost/requirement  trade  studies  are  used  to 
support  operational  requirements  scrubbing. 
These  trades  examine  each  requirement  and 
determine  the  cost  to  achieve  various  levels  of 
the  requirement  (e.g.,  different  airspeeds,  range, 
payloads).  The  results  are  then  used  to  determine, 
with  the  user,  whether  a  particular  requirement 
level  is  worth  the  cost  of  achieving  that  level. 
Trade  studies  are  an  inherent  part  of  the  systems 
engineering  process.  (See  Deskbook  2.6.1  for 
details  on  systems  engineering  process.) 

5.6.4  Risk  Assumption 

5.6.4. 1  Description.  This  technique  is  used  in 
every  program  and  acknowledges  the  fact  that, 
in  any  program,  risks  exist  that  will  have  to  be 
accepted  without  any  special  effort  to  control 
them.  Such  risks  may  be  either  inherent  in  the 
program  or  may  result  from  other  risk-control- 
ling  actions  (residual  risks).  The  fact  that  risks 
are  assumed  does  not  mean  that  they  are 
ignored.  In  fact,  every  effort  should  be  made  to 
identify  and  understand  them  so  that  appropri¬ 
ate  management  action  can  be  planned.  Also, 
risks  that  are  assumed  should  be  monitored  dur¬ 
ing  development;  this  monitoring  should  be  well- 
planned  from  the  beginning. 

5.6.4.2  Procedures.  In  addition  to  the  identifi¬ 
cation  of  risks  to  be  assumed,  the  following  steps 
are  key  to  successful  risk  assumption: 


•  Identify  the  resources  (time,  money,  people, 
etc.)  needed  to  overcome  a  risk  if  it  material¬ 
izes.  This  includes  identifying  the  specific 
management  actions  that  will  be  used,  for 
example,  redesign,  retesting,  requirements 
review,  etc. 

•  Whenever  a  risk  is  assumed,  a  schedule  and 
cost  risk  reserve  should  be  set  aside  to  cover 
the  specific  actions  to  be  taken  if  the  risk 
occurs.  If  this  is  not  possible,  the  program 
may  proceed  within  the  funds  and  schedule 
allotted  to  the  effort.  If  the  program  cannot 
achieve  its  objectives,  a  decision  must  be  made 
to  allocate  additional  resources,  accept  a 
lower  level  of  capability  (lower  the 
requirements),  or  cancel  the  effort. 

•  Ensure  that  the  necessary  administrative  actions 
are  taken  to  quickly  report  on  the  risk  event 
and  implement  these  management  actions,  such 
as  contracts  for  industry  expert  consultants,  ar¬ 
rangements  for  test  facilities,  etc.,  and  report 
on  occurrences  of  the  risk  event. 

5.6.5  Risk  Transfer 

5.6.5. 1  Description.  This  technique  involves  the 
reduction  of  risk  exposure  by  the  reallocation  of 
risk  from  one  part  of  the  system  to  another  or  the 
reallocation  of  risks  between  the  Government 
and  the  prime  contractor,  or  between  the  prime 
contractor  and  its  sub-contractor. 

5.6.5.2  Procedures.  In  reallocating  risk,  design 
requirements  that  are  risk  drivers  are  transferred 
to  other  system  elements,  which  may  result  in 
lower  system  risk  but  still  meet  system  require¬ 
ments.  For  example,  a  high  risk  caused  by  a  sys¬ 
tem  timing  requirement  may  be  lowered  by  trans¬ 
ferring  that  requirement  from  a  software  module 
to  a  specially  designed  hardware  module  capable 
of  meeting  those  needs.  The  effectiveness  of  re¬ 
quirements  reallocation  depends  on  good  sys¬ 
tem  engineering  and  design  techniques.  In  fact, 
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efficient  allocation  of  those  requirements  that  are 
risk  drivers  is  an  integral  part  of  the  systems  en¬ 
gineering  process.  Modularity  and  functional  par¬ 
titioning  are  two  design  techniques  that  can  be 
used  to  support  this  type  of  risk  transfer.  In  some 
cases,  this  approach  may  be  used  to  concentrate 
risk  areas  in  one  area  of  the  system  design.  This 
allows  management  to  focus  attention  and  re¬ 
sources  on  that  area. 

For  the  Government/contractor  risk-transfer 
approach  to  be  effective,  the  risks  transferred  to 
the  contractor  must  be  those  that  the  contractor 
has  the  capacity  to  control  and  manage.  These 
are  generally  risks  associated  with  technologies 
and  processes  used  in  the  program  — those  for 
which  the  contractor  can  implement  proactive 
solutions.  The  types  of  risks  that  are  best  man¬ 
aged  by  the  Government  include  those  related 
to  the  stability  of  and  external  influences  on  pro¬ 
gram  requirements,  funding,  and  schedule,  for 
example.  The  contractor  can  support  the  man¬ 
agement  of  these  risks  through  the  development 
of  flexible  program  plans,  and  the  incorporation 
of  performance  margins  in  the  system  and  flex¬ 
ibility  in  the  schedule.  A  number  of  options  are 
available  to  implement  risk  transfer  from  the 
Government  to  the  contractor:  warranties,  cost 
incentives,  product  performance  incentives,  and 
various  types  of  fixed  price  contracts.  A  similar 
assessment  of  prime  contractor  versus  sub-con¬ 
tractor  allocation  of  risks  can  also  be  developed 
and  used  to  guide  risk  transfer  between  these 
parties. 

5.7  RISK  MONITORING 
5.7.1  General 

Risk  monitoring  is  a  continuous  process  to 
systematically  track  and  evaluate  the  perfor¬ 
mance  of  risk-handling  actions  against  estab¬ 
lished  metrics  throughout  the  acquisition  process. 
It  should  also  include  results  of  periodic  reas¬ 
sessments  of  program  risk  to  evaluate  both 


known  and  new  risks  to  the  program.  If  neces¬ 
sary,  the  PMO  should  reexamine  the  risk-han¬ 
dling  approaches  for  effectiveness  while  con¬ 
ducting  assessments.  As  the  program  progresses, 
the  monitoring  process  will  identify  the  need  for 
additional  risk-handling  options. 

An  effective  monitoring  effort  provides  infor¬ 
mation  to  show  if  handling  actions  are  not 
working  and  which  risks  are  on  their  way  to  be¬ 
coming  actual  problems.  The  information  should 
be  available  in  sufficient  time  for  the  PMO  to 
take  corrective  action.  The  functioning  of  IPTs 
is  crucial  to  effective  risk  monitoring.  They  are 
the  “front  line”  for  obtaining  indications  that  han¬ 
dling  efforts  are  achieving  their  desired  effects. 

The  establishment  of  a  management  indicator 
system  that  provides  accurate,  timely,  and 
relevant  risk  information  in  a  clear,  easily 
understood  manner  is  key  to  risk  monitoring. 
Early  in  the  planning  phase  of  the  process,  PMOs 
should  identify  specific  indicators  to  be 
monitored  and  information  to  be  collected,  com¬ 
piled,  and  reported.  Usually,  documentation  and 
reporting  procedures  are  developed  as  part  of 
risk  management  planning  before  contract  award 
and  should  use  the  contractor’s  reporting  sys¬ 
tem.  Specific  procedures  and  details  for  risk  re¬ 
porting  should  be  included  in  the  risk  manage¬ 
ment  plans  prepared  by  the  Government  and  the 
contractor. 

To  ensure  that  significant  risks  are  effectively 
monitored,  handling  actions  (which  include  spe¬ 
cific  events,  schedules,  and  “success”  criteria) 
developed  during  previous  risk  management 
phases  should  be  reflected  in  integrated  program 
planning  and  scheduling.  Identifying  these  han¬ 
dling  actions  and  events  in  the  context  of  WBS 
elements  establishes  a  linkage  between  them  and 
specific  work  packages,  making  it  easier  to  de¬ 
termine  the  impact  of  actions  on  cost,  schedule, 
and  performance.  The  detailed  information  on 
risk-handling  actions  and  events  should  be 
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contained  in  various  risk  management  documen¬ 
tation  (both  formal  and  informal).  Experience  has 
shown  that  the  use  of  an  electronic  on-line  data¬ 
base  that  stores  and  permits  retrieval  of  risk-re¬ 
lated  information  is  almost  essential  to  effective 
risk  monitoring.  The  database  selected  or  devel¬ 
oped  will  depend  on  the  program.  A  discussion 
of  risk  management  information  systems  and 
databases  and  suggested  data  elements  to  be  in¬ 
cluded  in  the  databases  is  contained  later  in  this 
chapter. 

5.7.2  Earned  Value  Management 

5.7.2. 1  Description.  Earned  value  (EV)  is  a 
management  technique  that  relates  resource  plan¬ 
ning  to  schedules  and  to  technical  performance 
requirements.  It  is  useful  in  monitoring  the  ef¬ 
fectiveness  of  risk-handling  actions  in  that  it  pro¬ 
vides  periodic  comparisons  of  the  actual  work 
accomplished  in  terms  of  cost  and  schedule  with 
the  work  planned  and  budgeted.  These  compari¬ 
sons  are  made  using  a  performance  baseline  that 
is  established  by  the  contractor  and  the  PM  at 
the  beginning  of  the  contract  period.  This  is  ac¬ 
complished  through  the  Integrated  Baseline  Re¬ 
view  (IBR)  process.  The  baseline  must  capture 
the  entire  technical  scope  of  the  program  in  de¬ 
tailed  work  packages.  The  baseline  also  includes 
the  schedule  to  meet  the  requirements  as  well  as 
the  resources  to  be  applied  to  each  work  pack¬ 
age.  Specific  risk-handling  actions  should  be  in¬ 
cluded  in  these  packages.  See  Defense  Acquisi¬ 
tion  Deskbook  Section  2.B.2.1  for  a  more  de¬ 
tailed  discussion  of  Earned  Value  and  the  IBR. 

5.7.2.2  Procedures.  The  periodic  EV  data  can 
provide  indications  of  risk  and  the  effectiveness 
of  handling  actions.  When  variances  in  cost  or 
schedule  begin  to  appear  in  work  packages 
containing  risk-handling  actions,  or  in  any  work 
package,  the  appropriate  IPTs  can  analyze  the 
data  to  isolate  causes  of  the  variances  and  gain 
insights  into  the  need  to  modify  or  create 
handling  actions. 


5.7.3  Technical  Performance 
Measurement 

5.7.3. 1  Description.  Technical  performance 
measurement  (TPM)  is  a  technique  that  com¬ 
pares  estimated  values  of  key  performance 
parameters  with  achieved  values,  and  determines 
the  impact  of  any  differences  on  system  effec¬ 
tiveness.  This  technique  can  be  useful  in  risk 
monitoring  by  comparing  planned  and  achieved 
values  of  parameters  in  areas  of  known  risk.  The 
periodic  application  of  this  technique  can  pro¬ 
vide  early  and  continuing  predictions  of  the  ef¬ 
fectiveness  of  risk-handling  actions  or  the  de¬ 
tection  of  new  risks  before  irrevocable  impacts 
on  the  cost  or  schedule  occur. 

5.7.3.2  Procedures.  The  technical  performance 
parameters  selected  should  be  those  that  are  in¬ 
dicators  of  progress  in  the  risk-handling  action 
employed.  They  can  be  related  to  system  hard¬ 
ware,  software,  human  factors,  and  logistics — 
any  product  or  functional  area  of  the  system.  Pa¬ 
rameter  values  to  be  achieved  through  the 
planned  handling  action  are  forecast  in  the  form 
of  planned  performance  profiles.  Achieved  val¬ 
ues  for  these  parameters  are  compared  with  the 
expected  values  from  the  profile,  and  any  differ¬ 
ences  are  analyzed  to  get  an  indication  of  the 
effectiveness  of  the  handling  action.  For  ex¬ 
ample,  suppose  a  system  requires  the  use  of  a 
specific  technology  that  is  not  yet  mature  and 
the  use  of  which  has  been  assessed  as  high  risk. 
The  handling  technique  selected  is  risk  control, 
and  an  off-line  technology  maturation  effort  will 
be  used  to  get  the  technology  to  the  level  where 
the  risk  is  acceptable.  The  technology  is  ana¬ 
lyzed  to  identify  those  parameters  that  are  key 
drivers,  and  performance  profiles  that  will  result 
from  a  sufficiently  mature  technology  are  estab¬ 
lished.  As  the  maturation  effort  progresses,  the 
achieved  values  of  these  parameters  are  com¬ 
pared  with  the  planned  profile.  If  the  achieved 
values  meet  the  planned  profile,  it  is  an  indicator 
that  the  risk-handling  approach  is  progressing 
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satisfactorily;  if  the  achieved  values  fall  short  of 
the  expected  values,  it  is  an  indicator  that  the 
approach  is  failing  to  meet  expectations  and  cor¬ 
rective  action  may  be  warranted. 

5.7.4  Integrated  Planning  and  Scheduling 

5.7.4. 1  Description.  Once  a  contract  has  been 
awarded,  techniques  such  as  integrated  planning 
and  scheduling  (integrated  master  plans  (IMP) 
and  integrated  master  schedules  (IMS))  can  be¬ 
come  invaluable  program  baseline  and  risk¬ 
monitoring  tools.  Integrated  planning  identifies 
key  events,  milestones,  reviews,  all  integrated 
technical  tasks,  and  risk-reduction  actions  for  the 
program,  along  with  accomplishment  criteria  to 
provide  a  definitive  measure  that  the  required 
maturity  or  progress  has  been  achieved.  Inte¬ 
grated  scheduling  describes  the  detailed  tasks 
that  support  the  significant  activities  identified 
in  integrated  planning  and  timing  of  tasks.  Also, 
the  integrated  schedule  can  include  the  resources 
planned  to  complete  the  tasks.  The  events,  tasks, 
and  schedule  resulting  from  integrated  planning 
are  linked  with  contract  specification  require¬ 
ments,  WBS,  and  other  techniques  such  as  TPM. 
When  the  events  and  tasks  are  related  to  risk- 
reduction  actions,  this  linkage  provides  a  signifi¬ 
cant  monitoring  tool,  giving  specific  insights  into 
the  relationships  among  cost,  schedule,  and  per¬ 
formance  risks. 

5.7.4.2  Procedures.  In  integrated  planning,  the 
Government  and  contractor  (or  other  perform¬ 
ing  activity)  should  identify  key  activities  of  the 
program,  to  include  risk-handling  actions  and 
success  criteria.  The  contractor  should  then 
prepare  the  integrated  schedule  reflecting  the 
planned  completion  of  tasks  associated  with  these 
activities.  As  the  program  progresses,  the  PMO 
can  monitor  effectiveness  of  handling  activities 
included  in  the  integrated  planning  events  and 
schedule  by  comparing  observed  activity  results 
with  their  criteria  and  determining  any  devia¬ 
tions  from  the  planned  schedule.  Any  failures  of 


handling  actions  to  meet  either  the  event  criteria 
or  schedule  should  be  analyzed  to  determine  the 
deviation’s  impact,  causes,  and  need  for  any 
modifications  to  the  risk-handling  approach. 

5.7.5  Watch  List 

5.7.5. 1  Description.  The  watch  list  is  a  listing 
of  critical  areas  which  management  should  pay 
special  attention  to  during  program  execution.  It 
is  a  straightforward,  easily  prepared  document 
that  is  derived  from  a  prioritized  list  of  risks.  It 
may  include  such  things  as  the  priority  of  the 
risk,  how  long  it  has  been  on  the  watch  list,  han¬ 
dling  actions,  planned  and  actual  completion 
dates  for  handling  actions,  and  explanations  for 
any  differences.  See  Table  5-3  for  an  example 
watch  list. 

5.7.5.2  Procedures.  Watch  list  development  is 
based  on  the  results  of  the  risk  assessment.  It  is 
common  to  keep  the  number  of  risks  on  the  watch 
list  relatively  small,  focusing  on  those  that  can 
have  the  greatest  impact  on  the  program.  Items 
can  be  added  as  the  program  unfolds  and  peri¬ 
odic  reassessments  are  conducted.  If  a  consider¬ 
able  number  of  new  risks  are  significant  enough 
to  be  added  to  the  watch  list,  it  may  be  an  indi¬ 
cator  that  the  original  assessment  was  not  accu¬ 
rate  and  that  program  risk  is  greater  than  initially 
thought.  It  may  also  indicate  that  the  program  is 
on  the  verge  of  becoming  out  of  control.  If  a  risk 
has  been  on  the  watch  list  for  a  long  time  be¬ 
cause  of  a  lack  of  risk-handling  progress,  a  reas¬ 
sessment  of  the  risk  or  the  handling  approach 
may  be  necessary.  Items  on  the  watch  list  should 
be  reviewed  during  the  various  program  reviews/ 
meetings,  both  formal  and  informal. 

5.7.6  Reports 

5.7.6. 1  Description.  Reports  are  used  to  con¬ 
vey  information  to  decision  makers  and  program 
team  members  on  the  status  of  risks  and  the  ef¬ 
fectiveness  of  risk-handling  actions.  Risk-related 
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Potential  Risk  Area 

Risk  Reduction  Actions 

Action  Code 

Due  Date 

Date  Completed 

Explanation 

•  Accurately 
predicting  shock 
environment 
shipboard 
equipment  will 
experience. 

•  Use  multiple  finite 
element  codes  & 
simplified  numerical 
models  for  early 
assessments. 

•  Shock  test  simple 
isolated  deck,  and 
proposed  isolated 
structure  to  improve 
confidence  in 
predictions. 

SEA  03P31 

SEA  03P31 

31  Aug  01 

31  Aug  02 

•  Evaluating 
acoustic  impact 
of  the  ship 
systems  that  are 
not  similar  to 
previous  designs. 

•  Concentrate  on 
acoustic  modeling 
and  scale  testing  of 
technologies  not 
demonstrated 
successfully  in  large- 
scale  tests  or  full- 
scale  tests. 

SEA  03TC 

31  Aug  01 

•  Factor  acoustic 
signature  mitigation 
from  isolated  modular 
decks  into  system 
requirements. 

Continue  model  tests 
to  validate  predictions 
for  isolated  decks. 

SEA  03TC 

31  Aug  02 

Table  5-3.  Watch  List  Example 


reports  can  be  presented  in  a  variety  of  ways, 
ranging  from  informal  verbal  reports  when  time 
is  of  the  essence  to  formal  summary-type  reports 
presented  at  milestone  reviews.  The  level  of  de¬ 
tail  presented  will  depend  on  the  audience. 

5.7.6.2  Procedures.  Successful  risk  manage¬ 
ment  programs  include  timely  reporting  of  results 
of  the  monitoring  process.  Reporting 
requirements  and  procedures,  to  include  format 
and  frequency,  are  normally  developed  as  part 
of  risk  management  planning  and  are  docu¬ 
mented  in  the  risk  management  plan.  Reports 
are  normally  prepared  and  presented  as  part  of 
routine  program  management  activities.  They  can 
be  effectively  incorporated  into  program  man¬ 
agement  reviews  and  technical  milestones  to  in¬ 


dicate  any  technical,  schedule,  and  cost  barriers 
to  the  program  objectives  and  milestones  being 
met.  One  example  of  a  status  presentation  is 
shown  in  Figure  5-15.  It  shows  some  top-level 
risk  information  that  can  be  useful  to  the  PMO 
as  well  as  others  external  to  the  program. 

Although  this  level  of  reporting  can  provide 
quick  review  of  overall  risk  status  for  identified 
problems,  more  detailed  risk  planning  and  sta¬ 
tus  can  be  provided  on  individual  risk  items.  For 
example,  some  program  IPT s  have  combined  risk 
level  and  scheduled  activities  to  provide  a  graphi¬ 
cal  overview  of  risk  status  for  either  internal  or 
external  review.  One  method  for  graphically 
showing  risk  status  for  an  individual  item  is 
shown  in  Figure  5-16. 
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Risk 
Plan  # 

98-12-9 


Risk  Management  Status 


Risk  Issue 

Non-stock  Listed  Spares 


High  Moderate  Low 

d3d3C3} 


Status/Comment 

Data  still  in  review;  need  to 
assign  part  numbers. 


98-12-10  Engineering  Updates 

98-12-11  Spares  &  Support 

98-12-12  Long  Lead  Requisitions 

98-12-13  T.O.  Validation 

98-12-14  Lack  of  LSA  Records  for 
GFE* 

98-12-15  Program  Parts  Obsolescence 
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— -^^CiosecT) 
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Cl 
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-X 
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Data  reviewed;  updates  not 
required  at  this  time. 


Spares  listing  approved  in 
definitization  conference.  No 
current  abatement  plan. 

Closed  Issue. 

Contractor  LSA  plan 
submitted  for  approval; 
rescheduled  for  5/95. 

Analysis  in  work,  identifying 
last  opportunity  buys. 


98-12-51  Design  Maturity 
98-12-16  System  Y  Interface  Definition 


d 

3C 

^>£ciosecT) 

d 

3C 

Studying  Commercial  Mix 
Interface. 

Questions  about  antenna 
location  and  cable  raised  risk. 


(*  Detail  of  highlighted  item  described  in  Figure  5-16.) 


Figure  5-15.  Example  Showing  Detailed  List  of  Top-Level  Risk  Information 


5.7.7  Management  Indicator  System 

5.7.7.1  Description.  A  management  indicator 
system  is  a  set  of  indicators  or  metrics  that  pro¬ 
vide  the  PMO  with  timely  information  on  the 
status  of  the  program  and  risk-handling  actions, 
and  is  essential  to  risk  monitoring  and  program 
success.  To  be  meaningful,  these  metrics  should 
have  some  objective  value  against  which  ob¬ 
served  data  can  be  measured,  reflecting  trends 
in  the  program  or  lack  thereof.  Metrics  should 
be  developed  jointly  by  the  PMO  and  the  con¬ 
tractor.  The  contractor’s  approach  to  metrics 
should  be  a  consideration  in  the  proposal  evalu¬ 
ation  process.  If  the  contractor  does  not  have  an 
established  set  of  metrics,  this  may  be  an  area  of 
risk  that  will  need  to  be  addressed. 

5.7.7.2  Procedures.  Metrics  can  be  categorized 
as  relating  to  technical  performance,  cost,  and 
schedule.  Technical  performance  metrics  can  be 


further  broken  down  into  categories  such  as  en¬ 
gineering,  production,  and  support,  and  within 
these  groups  as  either  product-  or  process-related. 
Product-related  metrics  pertain  to  characteristics 
of  the  system  being  developed;  they  can  include 
such  things  as  planned  and  demonstrated  values 
of  the  critical  parameters  monitored  as  part  of 
the  TPM  process  and  system-unique  data  per¬ 
taining  to  the  different  steps  in  the  development 
and  acquisition  processes.  Table  5-4  provides  ex¬ 
amples  of  product-related  metrics. 

Process  metrics  pertain  to  the  various  processes 
used  in  the  development  and  production  of  the 
system.  For  each  program,  certain  processes  are 
critical  to  the  achievement  of  program  objectives. 
Failure  of  these  processes  to  achieve  their  re¬ 
quirements  is  symptomatic  of  significant  prob¬ 
lems.  Metrics  data  can  be  used  to  diagnose  and 
aid  in  problem  resolution.  They  should  be  used 
in  formal,  periodic  performance  assessments  of 
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Lack  of  Support  Records  for  GFE 


Action  Open  ^  -  Action  Completed 


Figure  5-16.  Example  of  More  Complex  Combination  of  Risk  Level  and  Scheduled  Tasks 


Engineering 

Requirements 

Production 

Support 

•  Key  Design 

Parameters 

-  Weight 

-  Size 

-  Endurance 

-  Range 

•  Design  Maturity 

-  Open  problems 
reports 

-  Number  of 
engineering  change 
proposals 

-  Number  of  drawings 
released 

-  Failure  activities 

•  Computer  Resource 

Utilization 

•  Requirements 
Traceability 

•  Requirements  Stability 

•  Manufacturing  Yields 

•  Incoming  Material 
Yields 

•  Delinquent 

Requisitions 

•  Unit  Production  Cost 

•  Process  Proofing 

•  Special  Tools  and  Test 
Equipment 

•  Support  Infrastructure 
Footprint 

•  Manpower  Estimates 

Table  5-4.  Examples  of  Product-Related  Metrics 
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the  various  development  processes  and  to  evalu¬ 
ate  how  well  the  system  development  process  is 
achieving  its  objectives.  DoD  4245. 7M,  Tran¬ 
sition  from  Development  to  Production,  and 
other  supporting  documents  such  as  NAVSO  P- 
607 1 ,  Best  Practices,  identify  seven  process  ar¬ 
eas:  funding,  design,  test,  production,  facilities, 
logistics,  and  management.  Within  each  of  these 
areas,  a  number  of  specific  processes  are  identi¬ 
fied  as  essential  to  assess,  monitor,  and  establish 
program  risk  at  an  acceptable  level;  the  docu¬ 
ments  also  provide  risk  indicators  that  can  be 
used  as  the  basis  for  selecting  specific  process 


metrics.  Another  document,  Methods  and 
Metrics  for  Product  Success,  July  1994,  pub¬ 
lished  by  the  Office  of  the  Assistant  Secretary 
of  the  Navy  (RD&A),  Product  Integrity  Direc¬ 
torate,  provides  a  set  of  metrics  for  use  in 
assessing  and  monitoring  the  design,  test,  and 
production  risk  areas.  Table  5-5  provides  ex¬ 
amples  of  process-related  metrics. 

Cost  and  schedule  metrics  can  be  used  to  depict 
how  the  program  is  progressing  toward  comple¬ 
tion.  The  information  provided  by  the  contrac¬ 
tor  in  the  earned  value  management  system  can 


Design 

Requirements 

Trade 

Studies 

Design 

Process 

Integrated  Test 
Plan 

Failure 

Reporting 

System 

Manufacturing 

Plan 

•  Development 
of  require¬ 
ments 
traceability 
plan 

•  Development 
of  specifica¬ 
tion  tree 

•  Specifications 
reviewed  for: 

-  Definition  of 
all  use 
environ¬ 
ments 

-  Definition  of 
all  func¬ 
tional 
require¬ 
ments  for 
each 
mission 
performed 

•  Users  needs 
prioritized 

•  Alternative 
system 
configura¬ 
tions  selected 

•  Test  methods 
selected 

•  Design 
requirements 
stability 

•  Producibility 
analysis 
conducted 

•  Design 
analyzed  for: 

-  Cost 

-  Parts 
reduction 

-  Manufac¬ 
turability 

-  Testability 

•  All  develop¬ 
mental  tests 
at  system 
and  sub¬ 
system  level 
identified 

•  Identification 
of  who  will  to 
test  (Govern¬ 
ment, 
contractor, 
supplier) 

•  Contractor 
corporate- 
level  manage¬ 
ment  involved 
in  failure 
reporting  and 
corrective 
action 
process 

•  Responsibility 
for  analysis 
and  corrective 
action 
assigned  to 
specific 
individual  with 
close-out  date 

•  Plan  docu¬ 
ments 
methods  by 
which  design 
to  be  built 

•  Plan  contains 
sequence  and 
schedule  of 
events  at 
contractor 
and  sub¬ 
contractor 
levels  that 
defines  use 
of  materials, 
fabrication 
flow,  test 
equipment, 
tools,  facili¬ 
ties,  and 
personnel 

•  Reflects 
manufactur¬ 
ing  inclusion 
in  design 
process. 
Includes 
identification 
and  assess¬ 
ment  of 
design 
facilities 

Table  5-5.  Examples  of  Process  Metrics 
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Cost 

Schedule 

Cost  variance 

Schedule  variance 

Cost  performance  index 

Schedule  performance  index 

Estimate  at  completion 

Design  schedule  performance 

Management  reserve 

Manufacturing  schedule  performance 

Test  schedule  performance 

Table  5-6.  Examples  of  Cost  and  Schedule  Metrics 


serve  as  these  metrics,  showing  how  the  actual 
work  accomplished  compares  with  the  work 
planned  in  terms  of  schedule  and  cost.  Other 
sources  of  cost  and  schedule  metrics  include  the 
contractor’s  cost  accounting  information  and  the 
integrated  master  schedule.  Table  5-6  provides 
examples  of  cost  and  schedule  metrics. 

5.8  RISK  MANAGEMENT 
INFORMATION  SYSTEMS 
AND  DOCUMENTATION 

5.8.1  Description 

To  manage  risk,  PMs  should  have  a  database 
management  system  that  stores  and  allows 
retrieval  of  risk-related  data.  The  risk- 
management  information  system  provides  data 
for  creating  reports  and  serves  as  the  repository 
for  all  current  and  historical  information  related 
to  risk.  This  information  may  include  risk  as¬ 
sessment  documents,  contract  deliverables,  if 
appropriate,  and  any  other  risk-related  reports. 
The  PM  should  consider  a  number  of  factors  in 
establishing  the  management  information  system 
and  developing  rules  and  procedures  for  the  re¬ 
porting  system: 

•  Assign  management  responsibility  for  the  re¬ 
porting  system; 

•  Publish  any  restrictions  for  entering  data  into 
the  database; 

•  Identify  reports  and  establish  a  schedule,  if 
appropriate; 


•  Use  standard  report  formats  as  much  as 
possible; 

•  Ensure  that  the  standard  report  formats 
support  all  users,  such  as  the  PM,  IPTs,  and 
IIPTs; 

•  Establish  policy  concerning  access  to  the  re¬ 
porting  system  and  protect  the  database  from 
unauthorized  access. 

With  a  well- structured  information  system,  a 
PMO  may  create  reports  for  senior  management 
and  retrieve  data  for  day-to-day  program 
management.  Most  likely,  the  PM  will  choose  a 
set  of  standard  reports  that  suits  specific  needs 
on  a  periodic  basis.  This  eases  definition  of  the 
contents  and  structure  of  the  database.  In  addi¬ 
tion  to  standard  reports,  the  PMO  will  need  to 
create  ad  hoc  reports  in  response  to  special  que¬ 
ries,  etc.  Commercial  database  programs  now 
available  allow  the  PMO  to  create  reports  with 
relative  ease.  Figure  5-17  shows  a  concept  for  a 
management  and  reporting  system. 

5.8.2  Risk  Management  Reports 

The  following  are  examples  of  basic  reports  that 
a  PMO  may  use  to  manage  its  risk  program.  Each 
office  should  tailor  and  amplify  them,  if  neces¬ 
sary,  to  meet  specific  needs. 

Risk  Information  Form  (RIF).  The  PMO 

needs  a  document  that  serves  the  dual  purpose 
of  a  source  of  data  entry  information  and  a  re¬ 
port  of  basic  information  for  the  IPTs.  The  RIF 
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Figure  5-1 7.  Conceptual  Risk  Management  and  Reporting  System 


serves  this  purpose.  It  gives  members  of  the 
project  team,  both  Government  and  contractors, 
a  format  for  reporting  risk-related  information. 
The  RIF  should  be  used  when  a  potential  risk 
event  is  identified  and  updated  over  time  as  in¬ 
formation  becomes  available  and  the  status 
changes.  As  a  source  of  data  entry,  the  RIF  allows 
the  database  administrator  to  control  entries.  To 
construct  the  database  and  ensure  the  integrity 
of  data,  the  PMO  should  design  a  standard  for¬ 
mat  for  a  RIF. 

Risk  Assessment  Report  (RAR).  Risk  assess¬ 
ments  form  the  basis  for  many  program  deci¬ 
sions,  and  the  PM  will  probably  need  a  detailed 
report  of  any  assessment  of  a  risk  event.  A  RAR 
is  prepared  by  the  team  that  assessed  a  risk  event 
and  amplifies  the  information  in  the  RIF.  It  docu¬ 
ments  the  identification  and  analysis  process  and 
results.  The  RAR  provides  information  for  the 
summary  contained  in  the  RIF,  is  the  basis  for 
developing  risk-handling  plans,  and  serves  as  a 
historical  recording  of  program  risk  assessment. 
Since  RARs  may  be  large  documents,  they  may 
be  stored  as  files.  RARs  should  include 
information  that  links  it  to  the  appropriate  RIF. 

Risk-Handling  Documentation.  Risk-handling 
documentation  may  be  used  to  provide  the  PM 


with  the  information  he  needs  to  choose  the  pre¬ 
ferred  handling  option  and  is  the  basis  for  the 
handling  plan  summary  that  is  contained  in  the 
RIF.  This  document  describes  the  examination 
process  for  the  risk-handling  options  and  gives 
the  basis  for  the  selection  of  the  recommended 
choice.  After  the  PM  chooses  an  option,  the 
rationale  for  that  choice  may  be  included.  There 
should  be  a  plan  for  each  risk-handling  task. 
Risk-handling  plans  are  based  on  results  of  the 
risk  assessment.  This  document  should  include 
information  that  links  it  to  the  appropriate  RIF. 

Risk  Monitoring  Documentation.  The  PM 

needs  a  summary  document  that  tracks  the  status 
of  high  and  moderate  risks.  He  can  produce  a 
risk-tracking  list,  for  example,  that  uses  infor¬ 
mation  that  has  been  entered  from  the  RIF.  Each 
PMO  should  tailor  the  tracking  list  to  suit  its 
needs.  If  elements  of  needed  information  are  not 
included  in  the  RIF,  they  should  be  added  to  that 
document  to  ensure  entry  into  the  database. 

Database  Management  System  (DBMS).  The 

DBMS  that  the  PM  chooses  may  be  commercial, 
Government-owned,  or  contractor-developed. 
It  should  provide  the  means  to  enter  and  access 
data,  control  access,  and  create  reports.  Many 
options  are  available  to  users. 
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Key  to  the  MIS  are  the  data  elements  that  reside 
in  the  database.  The  items  listed  in  Table  5-7  are 
examples  of  risk  information  that  might  be  in¬ 
cluded  in  a  database  that  supports  risk  manage¬ 
ment.  They  are  a  compilation  of  several  risk  re¬ 
porting  forms  used  in  current  DoD  programs  and 
other  risk  document  sources.  “Element”  is  the 
title  of  the  database  field;  “Description”  is  a  sum¬ 
mary  of  the  field  contents.  PMs  should  tailor  the 
list  to  suit  their  needs. 

5.9  SOFTWARE  RISK  MANAGEMENT 
METHODOLOGIES 

The  management  of  risk  in  software  intensive 
programs  is  essentially  the  same  as  for  any  other 
type  of  program.  A  number  of  methodologies 
specifically  focus  on  the  software  aspects  of 
developmental  programs  and  can  be  useful  in 
identifying  and  analyzing  risks  associated  with 
software.  Several  of  these  methodologies  are 
described  in  the  U.S.  Air  Force  publication, 
Guide  to  Software  Acquisition  and  Management. 
Three  of  these  methodologies  are  described  be¬ 
low. 

5.9.1  Software  Risk  Evaluation  (SRE) 

This  is  a  formal  approach  developed  by  the  Soft¬ 
ware  Engineering  Institute  (SEI)  using  a  risk 
management  paradigm  that  defines  a  continu¬ 
ous  set  of  activities  to  identify,  communicate, 
and  resolve  software  risks.  These  activities  are 
to  identify,  analyze,  plan,  track,  and  control. 
(The  SEI  activities  are  analogous  to  the  activi¬ 
ties  of  the  risk  management  process  defined  in 
this  section.) 

This  methodology  is  initiated  by  the  PM,  who  tasks 
an  independent  SRE  team  to  conduct  a  risk  evalu¬ 
ation  of  the  contractor’s  software  development  ef¬ 
fort.  The  team  executes  the  following  SRE 
functions  in  performing  this  evaluation,  and  pre¬ 
pares  findings  that  will  provide  the  PM  with  the 
results  of  the  evaluation: 


•  Detection  of  the  software  technical  risks 
present  in  the  program.  An  SEI  Taxonomy- 
Based  Questionnaire  is  used  to  ensure  that 
all  areas  of  potential  risk  are  identified.  This 
questionnaire  is  based  on  the  SEI  Software 
Development  Risk  Taxonomy,  which  pro¬ 
vides  a  systematic  way  of  organizing  and  elic¬ 
iting  risks  within  a  logical  framework. 

•  Specification  of  all  aspects  of  identified  tech- 
nical  software  risks,  including  their 
conditions,  consequences/impacts,  and 
source. 

•  Assessment  of  the  risks  to  determine  the  prob¬ 
ability  of  risk  occurrence  and  the  severity  of 
its  consequences/impacts. 

•  Consolidation  of  the  risk  data  into  a  concise 
format  suitable  for  decision  making. 

A  detailed  discussion  of  the  SRE  methodology 
is  found  in  Software  Engineering  Institute  Tech¬ 
nical  Report  CMU/SEI-94-TR- 1 9,  Software  Risk 
Evaluation  Model,  Version  1.0,  December  1994. 

5.9.2  Boehm’s  Software  Risk 
Management  Method 

This  risk  management  methodology,  developed 
by  Barry  W.  Boehm  and  described  in  IEEE  Soft¬ 
ware,  Software  Risk  Management:  Principles 
and  Practices,  January  1991,  consists  of  two 
primary  steps,  each  with  three  subordinate  steps. 
This  risk  management  stmcture  is  shown  in  Table 
5-8. 

Boehm  provides  a  number  of  techniques  that  can 
be  used  to  accomplish  each  of  the  steps  in  the 
methodology.  For  example,  to  assist  in  risk 
identification,  he  includes  the  top  10  top-level 
software  risks,  based  on  surveys  of  experienced 
software  project  managers.  These  risks  are 
shown  in  Table  5-9,  along  with  recommended 
techniques  to  manage  them.  Using  this  list  as  a 
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Element 

Description 

Risk  Identification 
(ID)  Number 

Identifies  the  risk  and  is  a  critical  element  of  information,  assuming  that  a 
relational  database  will  be  used  by  the  PMO.  (Construct  the  ID  number  to 
identify  the  organization  responsible  for  oversight.) 

Risk  Event 

States  the  risk  event  and  identifies  it  with  a  descriptive  name.  The  statement 
and  risk  identification  number  will  always  be  associated  in  any  report. 

Priority 

Reflects  the  importance  of  this  risk  priority  assigned  by  the  PMO  compared  to 
all  other  risks,  e.g.,  a  one  (1)  indicates  the  highest  priority. 

Data  Submitted 

Gives  the  date  that  the  RIF  was  submitted. 

Major  System/ 
Component 

Identifies  the  major  system/component  based  on  the  WBS. 

Subsystem/ 
Functional  Area 

Identifies  the  pertinent  subsystem  or  component  based  on  the  WBS. 

Category 

Identifies  the  risk  as  technical/performance  cost  or  schedule  or  combination  of 
these. 

Statement  of  Risk 

Gives  a  concise  statement  (one  or  two  sentences)  or  the  risk. 

Description  of 

Risk 

Briefly  describes  the  risk.  Lists  the  key  processes  that  are  involved  in  the 
design,  development,  and  production  of  the  particular  system  or  subsystem.  If 
technical/performance,  includes  how  it  is  manifested  (e.g.,  design  and 
engineering,  manufacturing,  etc.). 

Key 

Parameters 

Identifies  the  key  parameter,  minimum  acceptable  value,  and  goal  value,  if 
appropriate.  Identifies  associated  subsystem  values  required  to  meet  the 
minimum  acceptable  value  and  describes  the  principal  events  planned  to 
demonstrate  that  the  minimum  value  has  been  met. 

Assessment 

States  if  an  assessment  has  been  done.  Cites  the  Risk  Assessment  Report,  if 
appropriate. 

Analyses 

Briefly  describes  the  analysis  done  to  assess  the  risk.  Includes  rationale  and 
basis  for  results. 

Probability  of 
Occurrence 

States  the  likelihood  of  the  event  occurring,  based  on  definitions  in  the 
program’s  Risk  Management  Plan. 

Consequence 

States  the  consequence  of  the  event,  if  it  occurs,  based  on  definitions  in  the 
program’s  Risk  Management  Plan. 

Time  Sensitivity 

Estimates  the  relative  urgency  for  implementing  the  risk-handling  option. 

Other  Affected 

Areas 

If  appropriate,  identifies  any  other  subsystem  or  process  that  this  risk  affects. 

Risk  Handling 

Plans 

Briefly  describes  plans  to  mitigate  the  risk.  Refers  to  any  detailed  plans  that 
may  exist,  if  appropriate. 

Risk  Monitoring 
Activity 

Measures  using  metrics  for  tracking  progress  in  implementing  risk-handling 
plans  and  achieving  planned  results  for  risk  reduction. 

Status 

Briefly  reports  the  status  of  the  risk-handling  activities  and  outcomes  relevant 
to  any  risk  handling  milestones. 

Status  Due  Date 

Lists  date  of  the  status  report. 

Assignment 

Lists  individual  assigned  responsibility  for  handling  activities. 

Reported  By 

Records  name  and  phone  number  of  individual  who  reported  the  risk. 

Table  5-7.  Database  Management  System  Elements 
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Primary  Steps 

Secondary  Steps 

Description 

Risk  Assessment 

Risk  Identification 

•  Produces  lists  of  project  specific  risk 
events 

Risk  Analysis 

•  Assesses  probability  of  risk  event  and 
consequences 

•  Assesses  compound  risk  resulting  from 
risk  event  interaction 

Risk  Prioritization 

•  Produces  rank-ordered  list  of  identified 
and  analyzed  risk  events 

Risk  Control 

Risk  Management  Planning 

•  Produces  plan  for  addressing  each  risk 
event 

•  Integrates  individual  risk  event  plans 
with  each  other  and  the  overall  plan 

Risk  Resolution 

•  Establishes  the  environment  and 
actions  to  resolve  or  eliminate  risks 

•  Tracks  progress  in  resolving  risks 

Risk  Monitoring 

•  Provides  feedback  for  refining 
prioritization  and  plans 

Table  5-8.  Software  Risk  Management  Steps 


Risk 

Risk  Management  Techniques 

Personnel  Shortfalls 

Staffing  with  top  talent;  job  matching  team  building;  key  personnel 
agreements;  cross  training 

Unrealistic  schedules  and 
budgets 

Detailed  multisource  cost  and  schedule  estimation;  design-to-cost; 
incremental  development;  software  reuse;  requirements  scrubbing 

Developing  the  wrong  software 
functions 

Organizational  analysis;  mission  analysis;  operations  concept 
formulation;  user  surveys;  prototyping;  early  users’  manuals 

Developing  wrong  user  interface 

Task  analysis;  prototyping;  scenarios;  user  characterization 
(functionality,  style,  workload) 

Goldplating 

Requirements  scrubbing;  prototyping;  cost/benefit  analysis; 
design-to-cost 

Continuing  stream  of 
requirements  changes 

High  change  threshold;  information  hiding;  incremental 
development  (defer  changes  to  later  increments) 

Shortfalls  in  externally  furnished 
components 

Benchmarking;  inspections;  reference  checking;  compatibility 
analysis 

Shortfalls  in  internally  performed 
tasks 

Reference  checking;  pre-award  audits;  award-fee  contracts; 
competitive  design  or  prototyping;  team  building 

Real-time  performance  shortfalls 

Simulation;  benchmarking;  modeling;  prototyping;  instrumentation; 
tuning 

Straining  computer  science 
capabilities 

Technical  analysis;  cost-benefit  analysis;  prototyping;  reference 
checking 

Table  5-9.  Top  10  Software  Risks 
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starting  point,  managers  and  engineers  can  then 
develop  lists  of  lower-level  risks  to  be  assessed 
and  resolved. 

5.9.3  Best  Practices  Initiative 

Risk  Management  Method 

The  Software  Acquisition  Best  Practices  Initia¬ 
tive  was  instituted  in  1994  to  improve  and  re¬ 
structure  the  software  acquisition  management 
process  through  the  identification  of  effective 
practices  used  in  successful  software  develop¬ 
ments.  One  result  of  this  effort  was  the  publi¬ 
cation  of  the  Program  Manager’s  Guide  to  Soft¬ 
ware  Acquisition  Best  Practices  by  the  Software 
Program  Managers  Network  (SPMN).  This 
document  identified  nine  principal  best  practices 


that  are  essential  to  the  success  of  any  large-scale 
software  development.  The  first  of  these  nine  is 
formal  risk  management.  To  assist  in  implement¬ 
ing  this  top  practice,  SPMN  developed  a  three- 
part  methodology  consisting  of  the  following 
steps:  address  the  problem;  practice  essentials; 
and  check  status.  Specific  activities  associated 
with  these  steps  are  shown  in  Table  5-10. 

SPMN  provides  PMOs  with  specialized  train¬ 
ing  programs  covering  the  core  disciplines  and 
techniques  for  implementing  this  formal  risk 
management  practice,  as  well  as  the  other  best 
practices.  SPMN  also  has  available  (or  under 
development)  a  number  of  guidebooks  designed 
to  provide  software  developers  and  PMs  with 
practical  guidance  for  planning,  implementing, 


Best  Practices  Initiative  Risk  Management  Method 

Address  the 
Problem 

Practice  Essentials 

Check  Status 

•  Recognize  that  all 
software  has  risk 

•  Attempt  to  resolve 
risk  as  early  as 
possible  when  cost 
impact  is  less  than 
it  will  be  later  in 
development 

•  Identify  risks 

•  Decriminalize  risk 

•  Plan  for  risk 

•  Formally  designate  a  Risk  Officer 

•  Include  in  budget  and  schedule  a  risk 
reserve  buffer  of  time,  money,  and  other 
resources 

•  Compile  database  for  all  non-negligible 
risks 

•  Prepare  profile  for  each  risk  showing 
probability  and  consequences 

•  Include  all  risks  over  full  life  cycle 

•  Provide  frequent  risk  status  reports  that 
include: 

-  Top  10  risk  items 

-  Number  of  risk  items  resolved 

-  Number  of  new  risk  items 

-  Number  of  risk  items  unresolved 

-  Unresolved  risk  items  on  critical  path 

•  Probably  costs  for  unresolved  risks 

•  Risk  Officer  appointed? 

•  Risk  databases  set  up? 

•  Risk  assessments  have  clear 
impact  on  program  plans  and 
decisions? 

•  Frequency  and  timeliness  of  risk 
assessment  updates  consistent 
with  decision  updates? 

•  Objective  criteria  used  to  identify, 
assess,  and  manage  risk? 

•  Information  flow  patterns  and 
reward  criteria  support  identification 
of  risk  by  all  program  personnel? 

•  Risks  identified  throughout  entire 
life  cycle? 

•  Risk  management  reserve  exist? 

•  Risk  profile  for  every  risk,  and 
components  updated  regularly? 

•  Risk  management  plan  has  explicit 
provisions  for  altering  decision 
makers  when  risk  becomes 
imminent? 

Table  5-10.  Best  Practices  Initiative  Risk  Management  Method 
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and  monitoring  their  programs.  SPMN  can  be  the  resulting  risk  issues  across  the  studies  into 

accessed  on  the  Internet  at  http://spmn.com/.  six  categories  and  17  total  issues,  as  shown  in 

Table  5-11.  The  very  high  degree  of  overlap  be- 
In  addition  to  the  studies  by  Barry  Boehm,  and  tween  risk  issues  identified  in  the  10  underlying 

information  on  the  SPMN,  a  survey  was  con-  studies  suggest  that  some  risk  issues  are  corn- 
ducted  by  Conrow  and  Shishido  (See  Reference)  mon  to  many  software-intensive  projects, 

which  evaluated  10  prior  studies  and  categorized 


Risk  Grouping 

Software  Risk  Issue 

Project-Level 

1 .  Excessive,  immature,  unrealistic  or  unstable  requirements 

2.  Lack  of  involvement 

3.  Underestimation  of  project  complexity  or  dynamic  natures 

Project  Attributes 

4.  Performance  shortfalls  (includes  errors  and  quality) 

5.  Unrealistic  cost  or  schedule  (estimates  and/or  allocated  amounts) 

Management 

6.  Ineffective  project  management  (possible  at  multiple  levels) 

Engineering 

7.  Ineffective  integration,  assembly  and  test;  quality  control;  specialty 
engineering;  systems  engineering  or  (possible  at  multiple  levels) 

8.  Unanticipated  difficulties  associated  with  the  user  interface 

Work 

Environment 

9.  Immature  or  untried  design,  processes  or  technologies  selected 

10.  Inadequate  work  plans  or  configuration  control 

1 1 .  Inappropriate  methods  or  tool  selection  or  inaccurate  metrics 

Other 

12.  Poor  planning 

13.  Inadequate  or  excessive  documentation  or  review  process 

14.  Legal  or  contractual  issues  (e.g.,  litigation,  malpractice,  ownership) 

15.  Obsolescence  (includes  excessive  schedule  length) 

16.  Unanticipated  difficulties  with  subcontracted  items 

17.  Unanticipated  maintenance  and/or  support  costs 

Table  5-11.  Software  Risk  Grouping 
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APPENDIX  A 


DOD  RISK  MANAGEMENT 
POLICIES  AND  PROCEDURES 


DoD  policies  and  procedures  that  address  risk 
management  for  acquisition  programs  are  con¬ 
tained  in  five  key  documents: 

1 .  DoD  Directive  (DoDD)  5000. 1 ,  The  Defense 
Acquisition  System; 

2.  DoD  Instruction  (DoDI)  5000.2,  Operation 
of  the  Defense  Acquisition  System; 

3.  Interim  Defense  Acquisition  Guidebook 
(IDAG); 

4.  DoDD  5000.4,  OSD  Cost  Analysis  Improve¬ 
ment  Group;  and 

5.  DoD  Manual  5000. 4-M,  Cost  Analysis 
Guidance  and  Procedures. 

The  relevant  sections  of  each  document  are 
referenced  in  the  Defense  Acquisition  Deskbook 
under  Mandatory  Direction  and  are  displayed 
under  DoD-Wide  Practices.  They  present  strong 
statements  on  the  need  for  risk  management  but 
collectively  are  not  sufficient  to  enable  the 
establishment  of  an  effective  risk  management 
program.  The  following  are  verbatim  extracts  of 
sections  of  the  DoD  5000  series  of  documents 
that  address  risk  management  as  part  of  acquisi¬ 
tion  policy  and  procedures.  The  reader  should 
be  aware  that  changes  to  the  5000  series  could 
result  in  different  paragraph  numbers. 


1.  DoDD  5000.1  The  Defense  Acquisition 
System ,  12  May  2003 

Para  El.  6.  Cost  Sharing 

The  PM  shall  structure  the  acquisition  in  a  way 
that  neither  imposes  undue  risk  on  contractors, 
nor  requires  unusual  contractor  investment. 

Para  El. 14  . 

PMs  shall  reduce  technology  risk,  demonstrate 
technologies  in  a  relevant  environment,  and  iden¬ 
tify  technology  alternatives,  prior  to  program 
initiation.  They  shall  reduce  integration  risk  and 
demonstrate  product  design  prior  to  the  design 
readiness  review.  They  shall  reduce  manufac¬ 
turing  risk  and  demonstrate  producibility  prior 
to  full-rate  production. 

2.  DoD  Instruction  5000.2.  Operation  of  the 
Defense  Acquisition  System ,  12  May  2003 

Para  3.3. 2.1.  Spiral  Development 

In  this  process,  a  desired  capability  is  identified, 
but  the  end-state  requirements  are  not  known  at 
program  initiation.  Those  requirements  are  re¬ 
fined  through  demonstration  and  risk  manage¬ 
ment;  there  is  continuous  user  feedback;  and  each 
increment  provides  the  user  the  best  possible 
capability. 
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Para  3.4.2. 

Technologists  and  industry  shall  identify  and 
protect  promising  technologies  in  laboratories 
and  research  centers,  academia,  and  foreign  and 
domestic  commercial  sources;  reduce  the  risks 
of  introducing  these  technologies  into  the  acqui¬ 
sition  process;  and  promote  coordination,  coop¬ 
eration,  and  mutual  understanding  of  technol¬ 
ogy  issues. 

Para  3.5.3. 

The  AoA  shall  assess  the  critical  technologies 
associated  with  these  concepts,  including  tech¬ 
nology  maturity,  technical,  and,  if  necessary, 
technology  maturation  and  demonstration  needs. 

Para  3.6.1.  Purpose 

The  purpose  of  this  phase  (Technology  Devel¬ 
opment)  is  to  reduce  technology  risk  and  to  de¬ 
termine  the  appropriate  set  of  technologies  to  be 
integrated  into  a  full  system. 

Para  3.7.1. 1. 

The  purpose  of  the  SDD  phase  is  to  develop  a 
system  or  an  increment  of  capability;  reduce  in¬ 
tegration  and  manufacturing  risk  (technology 
risk  reduction  occurs  during  Technology  De¬ 
velopment);  ensure  operational  supportability 
with  particular  attention  to  reducing  the  logistics 
footprint;  implement  human  systems  integration 
(HSI);  design  for  producibility;  ensure 
affordability  and  the  protection  of  critical  pro¬ 
gram  information  (CPI)  by  implementing  appro¬ 
priate  techniques  such  as  anti-tamper;  and  dem¬ 
onstrate  system  integration,  interoperability, 
safety,  and  utility. 

Para  3. 7. 1.2. 

For  Shipbuilding  Programs,  the  required  program 
information  shall  be  updated  in  support  of  the 


Milestone  B  decision,  and  the  ICE  shall  be  com¬ 
pleted.  The  lead  ship  in  a  class  shall  normally  be 
authorized  at  Milestone  B.  Technology  readiness 
assessments  shall  consider  the  risk  associated 
with  critical  subsystems  prior  to  ship  installation. 

Para  3. 7.2.2. 

The  management  and  mitigation  of  technology 
risk,  which  allows  less  costly  and  less  time-con¬ 
suming  systems  development,  is  a  crucial  part 
of  overall  program  management  and  is  especially 
relevant  to  meeting  cost  and  schedule  goals. 
Objective  assessment  of  technology  maturity  and 
risk  shall  be  a  routine  aspect  of  DoD  acquisition. 

Para  3. 7.3.  System  Integration 

This  effort  is  intended  to  integrate  subsystems, 
complete  detailed  design,  and  reduce  system- 
level  risk. 

Para  3.7.4. 

The  Design  Readiness  Review  during  SDD  pro¬ 
vides  an  opportunity  for  mid-phase  assessment 
of  design  maturity  as  evidenced  by  measures  such 
as  the  number  of  subsystem  and  system  design 
reviews  successfully  completed;  the  percentage 
of  drawings  completed;  planned  corrective  ac¬ 
tions  to  hardware/software  deficiencies;  adequate 
development  testing;  an  assessment  of  environ¬ 
ment,  safety  and  occupational  health  risks;  a 
completed  failure  modes  and  effects  analysis;  the 
identification  of  key  system  characteristics  and 
critical  manufacturing  processes;  an  estimate  of 
system  reliability  based  on  demonstrated  reliabil¬ 
ity  rates;  etc. 

Para  3.8.2.  Entrance  Criteria 

Entrance  into  this  phase  (Production/Deploy¬ 
ment)  depends  on  the  following  criteria:  accept¬ 
able  performance  in  development,  test  and  evalu¬ 
ation  and  operational  assessment;  mature 
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software  capability;  no  significant  manufactur¬ 
ing  risks;  manufacturing  processes  under  con¬ 
trol  (if  Milestone  C  is  full-rate  production);  an 
approved  ICD  (if  Milestone  C  is  program  initia¬ 
tion);  an  approved  Capability  Production  Docu¬ 
ment  (CPD);  acceptable  interoperability;  accept¬ 
able  operational  supportability;  compliance  with 
the  DoD  Strategic  Plan;  and  demonstration  that 
the  system  is  affordable  throughout  the  life  cycle, 
optimally  funded,  and  properly  phased  for  rapid 
acquisition. 

Table  E3.T3.  Contract  Reporting 
Requirements 

The  CCDR  requirement  on  high-risk  or  high- 
technical-interest  contracts  priced  between  $7  and 
$50  million  is  left  to  the  discretion  of  the  Cost 
Working  Integrated  Product  Team  (IPT). 

Para  E5.1. 

The  T&E  strategy  shall  provide  information  about 
risk  and  risk  mitigation,  provide  empirical  data 
to  validate  models  and  simulations,  evaluate  tech¬ 
nical  performance  and  system  maturity,  and  de¬ 
termine  whether  systems  are  operationally  effec¬ 
tive,  suitable,  and  survivable  against  the  threat 
detailed  in  the  System  Threat  Assessment. 

Para  E5.3.1. 

Projects  that  undergo  a  Milestone  A  decision  shall 
have  a  T&E  strategy  that  shall  primarily  address 
M&S,  including  identifying  and  managing  the 
associated  risk,  and  that  shall  evaluate  system 
concepts  against  mission  requirements. 

Para  E5.4.6. 

The  concept  of  early  and  integrated  T&E  shall 
emphasize  prototype  testing  during  system  de¬ 
velopment  and  demonstration  and  early  OAs  to 
identify  technology  risks  and  provide  operational 
user  impacts. 


Para  E5.5.  Developmental  Test  and 
Evaluation  (DT&E) 

During  DT&E,  the  materiel  developer  shall  iden¬ 
tify  and  describe  design  technical  risks. 

Para  E5.6.1. 

The  process  shall  include  a  review  of  DT&E 
results;  an  assessment  of  the  system’s  progress 
against  critical  technical  parameters  documented 
in  the  TEMP;  an  analysis  of  identified  technical 
risks  to  verify  that  those  risks  have  been  retired 
during  developmental  testing;  and  a  review  of 
the  IOT&E  entrance  criteria  specified  in  the 
TEMP. 

Para  E6.4.  CAIG  Procedures 

The  DoD  Component  responsible  for  acquisi¬ 
tion  of  a  system  shall  cooperate  with  the  CAIG 
and  provide  the  cost,  programmatic,  and  techni¬ 
cal  information  required  for  estimating  costs  and 
appraising  cost  risks. 

Para  E6.5. 

In  this  evaluation,  the  D,PA&E  shall  assess  the 
extent  to  which  the  AoA: 

Assessed  technology  risk  and  maturity. 

Para  E7.7.  Environment,  Safety  and  Occu¬ 
pational  Health  (ESOH) 

As  part  of  risk  reduction,  the  PM  shall  prevent 
ESOH  hazards  where  possible,  and  shall  man¬ 
age  ESOH  hazards  where  they  cannot  be 
avoided.  The  acquisition  strategy  shall  incorpo¬ 
rate  a  summary  of  the  Programmatic  ESOH 
Evaluation  (PESHE),  including  ESOH  risks,  a 
strategy  for  integrating  ESOH  considerations  into 
the  systems  engineering  process,  identification 
of  ESOH  responsibilities,  a  method  for  tracking 
progress,  and  a  compliance  schedule  for  NEPA. 


For  acceptance  of  ESOH  mishap  risks  identi¬ 
fied  by  the  program,  the  CAE  is  the  acceptance 
authority  for  high  risks,  PEO-level  for  serious 
risks,  and  the  PM  for  medium  and  low  risks  as 
defined  in  the  industry  standard  for  system  safety. 

3.  Interim  Defense  Acquisition  Guidebook 
(IDAG),  30  October  2002 

Para  Cl. 3.4.2.  Management  Incentives 

The  PM,  via  the  Contracting  Officer,  shall  struc¬ 
ture  Requests  for  Proposal  (RFPs)  and  resulting 
contracts  to  provide  an  incentive  to  the  contrac¬ 
tor  to  meet  or  beat  program  objectives.  When¬ 
ever  applicable,  risk  reduction  through  use  of 
mature  processes  shall  be  a  significant  factor  in 
source  selection. . . . 

Para  Cl.4.3.3.2.  Cost 

Cost  figures  shall  reflect  realistic  estimates  of  the 
total  program,  including  a  thorough  assessment 
of  risk.... 

Para  C2.3.1.  Program  Structure 

...The  acquisition  strategy  shall  specifically 
address  the  benefits  and  risks  associated  with 
reducing  lead-time  through  concurrency  and  the 
risk  mitigation  and  tests  planned  if  concurrent 
development  is  used. . . . 

Para  C2.5.  Risk 

The  acquisition  strategy  shall  address  risk 
management.  The  PM  shall  identify  the  risk  ar¬ 
eas  of  the  program  and  integrate  risk  manage¬ 
ment  within  overall  program  management.  The 
strategy  shall  explain  how  the  risk  management 
effort  shall  reduce  system-level  risk  to  acceptable 
levels  by  the  interim  progress  review  preceding 
system  demonstration  and  by  Milestone  C. 


Para  C2.6.2.  Information  Sharing  and 
DoD  Oversight 

. .  .DoD  oversight  activities  (i.e.,  contract  man¬ 
agement  offices,  contracting  offices,  technical 
activities,  and  program  management  offices)  shall 
consider  all  relevant  and  credible  information  that 
might  mitigate  risk  and  reduce  the  need  for  DoD 
oversight  before  defining  and  applying  direct 
DoD  oversight  of  contractor  operations .... 

Para  C2.8.1.  Support  Strategy 

As  part  of  the  acquisition  strategy,  the  PM  shall 
develop  and  document  a  support  strategy  for  life- 
cycle  sustainment  and  continuous  improvement 
of  product  affordability,  reliability,  and  support- 
ability,  while  sustaining  readiness....  The  sup¬ 
port  strategy  shall  continue  to  evolve  toward 
greater  detail,  so  that  by  Milestone  C,  it  contains 
sufficient  detail  to  define  how  the  program  will 
address  the  support  and  fielding  requirements 
that  meet  readiness  and  performance  objectives, 
lower  TOC,  reduce  risks  and  avoid  harm  to  the 
environment  and  human  health.  The  support 
strategy  shall  address  all  applicable  support  re¬ 
quirements  to  include,  but  not  be  limited  to,  the 
following  elements: . . . 

Para  C2. 8. 1.7.3.  Support  Strategy 

Contract  service  risk  assessments  over  the  life  of 
the  system. 

Para  C2.8.4.2.2.  Supply  Source  of  Support 

The  PM  shall  use  a  competitive  process  to  select 
the  best  value  supply  support  provider.  Access 
to  multiple  sources  of  supply  is  encouraged  to 
reduce  the  risks  associated  with  a  single 
source.... 
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Para  C2.8.6.  Environment,  Safety,  and 
Occupational  Health  (ESOH) 
Considerations 

As  part  of  risk  reduction,  the  PM  shall  prevent 
ESOH  hazards,  where  possible,  and  shall  man¬ 
age  ESOH  hazards  where  they  cannot  be 
avoided.  The  support  strategy  shall  contain  a 
summary  of  the  Programmatic  ESOH  Evalua¬ 
tion  (PESHE)  document,  including  ESOH 
risks,  a  strategy  for  integrating  ESOH  consider¬ 
ations  into  the  systems  engineering  process,  iden¬ 
tification  of  ESOH  responsibilities,  a  method  for 
tracking  progress,  and  a  compliance  schedule  for 
the  National  Environmental  Policy  Act  (NEPA) 
and  Executive  Order  (E.O.)  12114. 

Para  C2.8.9.  Post  Deployment  Evaluation 

The  PM  shall  select  the  parameters  for  evalua¬ 
tions  based  on  their  relevance  to  future 
modifications  or  evolutionary  block  upgrades  for 
performance,  sustainability,  and  affordability 
improvements,  or  when  there  is  a  high  level  of 
risk  that  a  KPP  will  not  be  sustained  over  the 
life  of  the  system.... 

Para  C2. 9.1. 3.2.3  Sub-Tier  Competition 

During  early  exchanges  of  information  with  in¬ 
dustry  (e.g.,  the  draft  request  for  proposal  pro¬ 
cess),  PMs  shall  identify  the  critical  product  and 
technology  areas  that  the  primes  plan  to  provide 
internally  or  through  exclusive  teaming.  The  PM 
shall  assess  the  possible  competitive  effects  of 
these  choices.  The  PM  shall  take  action  to  miti¬ 
gate  areas  of  risk.... 

Para  C2.9.1.4.2.2  Commercial  and  Non- 
Developmental  Items 

. .  .If  acquiring  products  with  closed  interfaces, 
the  PM  shall  conduct  a  business  case  analysis  to 
justify  acceptance  of  the  associated  economic 
impacts  on  TOC  and  risks  to  technology  inser¬ 


tion  and  maturation  over  the  service  life  of  the 
system. 

Para  C2.9.1.4.4.1  Industrial  Capability 

The  acquisition  strategy  shall  summarize  an 
analysis  of  the  industrial  base  capability  to  de¬ 
sign,  develop,  produce,  support,  and,  if  appro¬ 
priate,  restart  the  program  (10  U.S.C.  2440  (ref¬ 
erence  (an)))  as  appropriate  for  the  next  program 
phase.  This  analysis  (see  DoD  Directive  5000.60 
(reference  (ao))  and  DoD  5000.60-H  (reference 
(ap)))  shall  identify  DoD  investments  needed  to 
create  or  enhance  certain  industrial  capabilities, 
and  the  risk  of  industry  being  unable  to  provide 
program  design  or  manufacturing  capabilities  at 
planned  cost  and  schedule. . . . 

Para  C2.9.1.4.4.2  Industrial  Capability 

In  many  cases,  commercial  demand  now  sus¬ 
tains  the  national  and  international  technology 
and  industrial  base.  The  PM  shall  structure  the 
acquisition  strategy  to  promote  sufficient  pro¬ 
gram  stability  to  encourage  industry  to  invest, 
plan,  and  bear  risks .... 

Para  C2.9.3.2.  Contract  Type 

For  each  major  contract,  the  acquisition  strategy 
shall  identify  the  type  of  contract  planned  (e.g., 
firm  fixed-price  (FFP);  fixed  price  incentive,  firm 
target;  cost  plus  incentive  fee;  or  cost  plus  award 
fee)  and  the  reasons  it  is  suitable,  including  con¬ 
siderations  of  risk  assessment  and  reasonable 
risk-sharing  by  the  Government  and  the 
contractor(s).... 

Para  C2.9.3.5.  Integrated  Baseline 
Reviews 

PMs  and  their  technical  staffs  or  IPTs  shall  evalu¬ 
ate  contract  performance  risks  inherent  in  the 
contractor’s  planning  baseline.  This  evaluation 
shall  be  initiated  within  6  months  after  contract 


award  or  intra-Govemment  agreement  is  reached 
for  all  contracts  requiring  EVMS  or  C/SSR 
compliance. 

Para  C2.9.3.8.  Component  Breakout 

The  PM  shall  consider  component  breakout  on 
every  program  and  break  out  components  when 
there  are  significant  cost  savings  (inclusive  of 
Government  administrative  costs),  the  technical 
or  schedule  risk  of  furnishing  government  items 
to  the  prime  contractor  is  manageable,  and  there 
are  no  other  overriding  Government  interests 
(e.g.,  industrial  capability  considerations  or  de¬ 
pendence  on  contractor  logistics  support). . . . 

Para  C3.1.1.  Test  and  Evaluation  (T&E) 
Overview 

. .  .The  T&E  strategy  shall  provide  information 
about  risk  and  risk  mitigation,  provide  empirical 
data  to  validate  models  and  simulations,  evaluate 
technical  performance  and  system  maturity,  and 
determine  whether  systems  are  operationally  ef¬ 
fective,  suitable,  and  survivable  against  the  threat 
detailed  in  the  System  Threat  Assessment.  (See 
paragraph  C6.2.4) . . . . 

Para  C3.2.1.1.  Evaluation  Strategy 

. .  .The  evaluation  strategy  shall  primarily  address 
M&S,  including  identifying  and  managing  the  as¬ 
sociated  risk,  and  early  T&E  strategy  to  evaluate 
system  concepts  against  mission  requirements .... 

Para  C3.2.3.2.1.  T&E  Guidelines 

Early  T&E  activities  shall  harmonize  MOEs, 
MOPs,  and  risk  with  the  needs  depicted  in  the 
MNS,  and  with  the  objectives  and  thresholds 
addressed  in  the  analysis  of  alternatives,  and 
defined  in  the  ORD,  APB,  and  TEMP,  as  these 
documents  become  available .... 


Para  C3.2.3.2.2.8.  T&E  Guidelines 

The  concept  of  early  and  integrated  T&E  shall 
emphasize  prototype  testing  during  system 
development  and  demonstration  and  early  OAs 
to  identify  technology  risks  and  provide 
operational  user  impacts .... 

Para  C3.4.1.2.  Developmental  Test  and 
Evaluation  (DT&E) 

Identify  and  describe  design  technical  risks. 
Assist  in  the  design  of  a  system  at  the  compo¬ 
nent,  sub-system,  and  system  level  by  reducing 
technical  risk  prior  to  transitioning  to  the  next 
level; 

Para  C3.4.1.6.  Developmental  Test  and 
Evaluation  (DT&E) 

Assess  progress  toward  meeting  KPPs  and 
other  ORD  requirements,  COIs,  mitigating  ac¬ 
quisition  technical  risk,  and  achieving 
manufacturing  process  requirements  and  sys¬ 
tem  maturity; 

Para  C3.5.1.  Certification  of  Readiness  for 
Operational  Test  &  Evaluation  (OT&E) 

The  developing  agencies  (i.e.,  materiel  and  com¬ 
bat  developers)  shall  complete  the  following  tasks 
before  starting  OT&E:  Define  risk  management 
measures  and  indicators,  with  associated  thresh¬ 
olds,  to  address  performance  and  technical  ad¬ 
equacy  of  both  hardware  and  software. 

Para  C3.6.1.3.  Operational  Test  and 
Evaluation  (OT&E) 

Information  assurance  testing  shall  be  conducted 
on  information  systems  to  ensure  that  planned 
and  implemented  security  measures  satisfy  ORD 
and  System  Security  Authorization  Agreement 
(SSAA)  requirements  when  the  system  is  in¬ 
stalled  and  operated  in  its  intended  environment. 
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The  PM,  OT&E  test  authority,  and  designated 
approving  authority  shall  coordinate  and  deter¬ 
mine  the  level  of  risk  associated  with  operating 
the  system  and  the  extent  of  security  testing  re¬ 
quired.  (See  section  C6.6.). . . 

Para  C3.6.1.13.  Operational  Test  and 
Evaluation  (OT&E) 

All  weapon,  Command,  Control,  Communica¬ 
tions,  Computers,  Intelligence,  Surveillance, 
and  Reconnaissance  (C4ISR),  and  information 
programs  that  are  dependent  on  external  infor¬ 
mation  sources,  or  that  provide  information  to 
other  DoD  systems,  shall  be  assessed  for  infor¬ 
mation  assurance.  The  level  of  information 
assurance  testing  depends  on  the  system  risk 
and  importance.  Systems  with  the  highest  im¬ 
portance  and  risk  shall  be  subject  to  penetra¬ 
tion-type  testing  prior  to  the  beyond  LRIP 
decision.  Systems  with  minimal  risk  and  im¬ 
portance  shall  be  subject  to  normal  National  Se¬ 
curity  Agency  security  and  developmental 
testing,  but  shall  not  be  subject  to  field 
penetration  testing  during  OT&E. 

ParaC4.3.  Analysis  of  Alternatives 

Analyzing  alternatives  is  part  of  the  CAIV 
process.  Alternatives  analysis  shall  broadly  ex¬ 
amine  multiple  elements  of  project  or  program 
alternatives  including  technical  risk  and 
maturity,  and  costs. 

Para  C4.5.1.2.  Resource  Estimates 

The  DoD  Component  cost  agency  shall  prepare 
an  independent  LCCE  and  associated  report  for 
the  decision  authority  for  all  ACAT IC  programs, 
except  those  reviewed  by  the  CAIG,  for  all  ma¬ 
jor  decision  points  as  specified  in  enclosure  3  of 
reference  (a),  or  as  directed  by  the  MDA.  For 
programs  with  significant  cost  risk  or  high  vis¬ 
ibility,  the  CAE  may  request  an  additional  DoD 
Component  cost  analysis  estimate. 


Para  C4.5.2.1.  Life- Cycle  Cost  Estimates 
(LCCEs) 

The  estimating  activity  shall  explicitly  base  the 
LCCE  (or  EA  for  ACAT  IA  programs)  on  pro¬ 
gram  objectives;  operational  requirements;  con¬ 
tract  specifications;  careful  risk  assessments;  and, 
for  ACAT  I  programs,  a  DoD  program  work 
breakdown  structure  (WBS),  or,  for  ACAT  IA 
programs,  a  life-cycle  cost  and  benefit  element 
structure  agreed  upon  by  the  IPT. . . . 

Para  C4.5.4.1.1.  Manpower 
Considerations 

For  all  programs  regardless  of  acquisition  category, 
the  DoD  Components  shall  determine  the  source 
of  support  for  all  new,  modified,  and  replacement 
systems  based  on  the  procedures,  manpower  mix 
criteria,  and  risk  assessment  instructions  in  Deputy 
Under  Secretary  of  Defense  (Program  Integration), 
Office  of  the  Under  Secretary  of  Defense  (Person¬ 
nel  &  Readiness)  (OUSD(P&R)),  and  Deputy 
Under  Secretary  of  Defense  (Installations),  Office 
of  USD(AT&L)  annual  memo,  “DoD  Inventory 
of  Commercial  and  Inherently  Governmental 
Activities  Data  Call . . . .” 

Para  C4. 5.4. 1.2.  Manpower 
Considerations 

The  DoD  Components  shall  determine  man¬ 
power  and  contract  support  based  on  both 
peacetime  and  wartime  requirements,  and  es¬ 
tablish  manpower  authorizations  at  the  mini¬ 
mum  necessary  to  achieve  specific  vital  objec¬ 
tives  (DoD  Directive  1 100.4  (reference  (bv))). 
As  part  of  this  process,  the  DoD  Components 
shall  assess  the  risks  (DoD  Instruction  3020.37 
(reference  (bw)))  involved  in  contracting  sup¬ 
port  for  critical  functions  in-theater,  or  in  other 
areas  expecting  hostile  fire.  Risk  mitigation 
shall  take  precedence  over  cost  savings  in  high- 
risk  situations  or  when  there  are  highly  sensi¬ 
tive  intelligence  or  security  concerns. 
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Para  C4. 5.4.2. 4.  Manpower  Estimate 

The  manpower  estimate  shall  address  whether 
there  are  any  personnel  issues  that  would  ad¬ 
versely  impact  full  operational  deployment  of  the 
system.  It  shall  clearly  state  the  risks  associated 
with  and  the  likelihood  of  achieving  manpower 
numbers  reported  in  the  estimate.  It  shall  briefly 
assess  the  validity  of  the  manpower  numbers, 
stating  whether  the  DoD  Component  used  vali¬ 
dated  manpower  methodologies  and  manpower 
mix  criteria,  and  assessed  all  risks .... 

Para  C5.2.1.  Systems  Engineering 

. .  .Systems  engineering  shall  permeate  design, 
manufacturing,  T&E,  and  support  of  the  product. 
Systems  engineering  principles  shall  influence 
the  balance  between  performance,  risk,  cost,  and 
schedule. 

Para  C5.2.2.3.  Systems  Engineering 

The  systems  engineering  process  shall. . .  Char¬ 
acterize  and  manage  technical  risks. 

Para  C5. 2.2.4.  Systems  Engineering 

The  systems  engineering  process  shall:  . . .  Apply 
scientific  and  engineering  principles,  using  the 
system  security  engineering  process,  to  identify 
security  vulnerabilities  and  minimize  or  contain 
information  assurance  and  force  protection  risks 
associated  with  these  vulnerabilities.  (See  DoD 
5200. 1-M  (reference  (bx)).) 

Para  C5.2.3.2.  Functional  Analysis/ 
Allocation 

. .  .The  design  approach  shall  partition  a  system 
into  self-contained,  functionally  cohesive, 
interchangeable,  and  adaptable  elements  to 
enable  ease  of  change,  achieve  technology  trans¬ 
parency  and  mitigate  risk  of  obsolescence. . . . 


Para  C5.2.3.4.2.  System  Analysis  and 
Control 

The  overall  risk  management  effort  shall  include 
technology  transition  planning  and  shall  estab¬ 
lish  transition  criteria. 

Para  C5.2.3.4.3.  System  Analysis  and 
Control 

The  establishment  of  a  risk  management  process 
(including  planning,  assessment  (identification 
and  analysis),  handling,  and  monitoring)  to  be 
integrated  and  continuously  applied  throughout 
the  program,  including,  but  not  limited  to,  the 
design  process.  The  risk  management  effort  shall 
address  risk  planning,  the  identification  and 
analysis  of  potential  sources  of  risks  including 
but  not  limited  to  cost,  performance,  and 
schedule  risks  based  on  the  technology  being 
used  and  its  related  design,  manufacturing  ca¬ 
pabilities,  potential  industry  sources,  and  test  and 
support  processes;  risk  handling  strategies,  and 
risk  monitoring  approaches.  The  overall  risk 
management  effort  shall  interface  with  technol¬ 
ogy  transition  planning,  including  the  establish¬ 
ment  of  transition  criteria  for  such  technologies. 

Para  C5.2.3.4.7.  System  Analysis  and 
Control 

Performance  metrics  to  measure  technical 
development  and  design,  actual  versus  planned; 
and  to  measure  meeting  system  requirements  in 
terms  of  performance,  cost,  schedule,  and 
progress  in  implementing  risk  handling.  Perfor¬ 
mance  metrics  shall  be  traceable  to  performance 
parameters  identified  by  the  operational  user. 

Para  C5.2.3. 5.5.2.10.  Open  Systems 

Design 

PMs  shall  use  an  open  systems  approach  to 
achieve  the  following  objectives: ...  To  mitigate 
the  risks  associated  with  technology 


obsolescence,  being  locked  into  proprietary 
technology,  and  reliance  on  a  single  source  of 
supply  over  the  life  of  a  system; 

Para  C5.2.3.5.6.  Software  Management 

The  PM  shall  manage  and  engineer  software¬ 
intensive  systems  using  best  processes  and 
practices  known  to  reduce  cost,  schedule,  and 
performance  risks. 

Para  C5. 2.3. 5. 6.1.3.  Software 
Management 

The  PM  shall  base  software  systems  design  and 
development  on  systems  engineering  principles, 
to  include  the  following:  . .  .Select  the  program¬ 
ming  language  in  context  of  the  systems  and 
software  engineering  factors  that  influence 
overall  life-cycle  costs,  risks,  and  the  potential 
for  interoperability; 

Para  C5. 2.3. 5. 6.1.5.  Software 
Management 

. .  .However,  if  the  prospective  contractor  does 
not  meet  full  compliance,  risk  mitigation  plan¬ 
ning  shall  describe,  in  detail,  the  schedule  and 
actions  that  will  be  taken  to  remove  deficiencies 
uncovered  in  the  evaluation  process.  Risk  miti¬ 
gation  planning  shall  require  PM  approval. 

Para  C5.2.3.5.6.1.7.  Software 
Management 

Assess  information  operations  risks  (DoD 
Directive  S-3600.1  (reference  (bz)))  using 
techniques  such  as  independent  expert  reviews; 

Para  C5.2.3.5.6.2.2.3.  Software  Spiral 
Development 

The  PM  shall  consider  the  risks  and  extent  of 
change  impacts  to  enable  a  cost-effective,  yet 
rigorous  T&E  process. 


Para  C5.2.3.5.6.4.7.  Software  Security 
Considerations 

When  employing  COTS  software,  the  contract¬ 
ing  process  shall  give  preference  during  prod¬ 
uct  selection/evaluation  to  those  vendors  who 
can  demonstrate  that  they  took  efforts  to  mini¬ 
mize  the  security  risks  associated  with  foreign 
nationals  that  have  developed,  modified,  or 
remediated  the  COTS  software  being  offered. 

Para  C5.2.3.5.7.2.5.  Commercial,  Off-the- 
Shelf  (COTS)  Considerations 

The  PM  shall  develop  an  appropriate  T&E  strat¬ 
egy  for  commercial  items  to  include  evaluating 
potential  commercial  items  in  a  system  test  bed, 
when  practical;  focusing  test  beds  on  high-risk 
items;  and  testing  commercial-item  upgrades  for 
unanticipated  side  effects  in  areas  such  as  secu¬ 
rity,  safety,  reliability,  and  performance. 

Para  C5.2.3.5.7.2.6.  Commercial,  Off-the- 
Shelf  (COTS)  Considerations 

Programs  are  encouraged  to  use  code-scanning 
tools,  within  the  scope  and  limitations  of  the  li¬ 
censing  agreements,  to  ensure  both  COTS  and 
Government  off-the-shelf  software  do  not  pose 
any  information  assurance  or  security  risks. 

Para  C5.2.3. 5.10.2.  Environment,  Safety, 
and  Occupational  Health  (ESOH) 

The  PM  shall  prepare  a  Programmatic  ESOH 
Evaluation  (PESHE)  document  early  in  the 
program  life  cycle  (usually  Milestone  B).  The 
PESHE  shall  identify  ESOH  risks,  contain  a 
strategy  for  integrating  ESOH  considerations 
into  the  systems  engineering  process,  delineate 
ESOH  responsibilities,  and  provide  a  method  for 
tracking  progress,  and  provide  a  completion 
schedule  for  NEPA  (reference  (x))  and  E.O. 
12114  (reference  (y)) . . . . 
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Para  C5.2.3.5.10.4.  ESOH  Compliance 

To  minimize  the  cost  and  schedule  risks  over 
the  system’s  life  cycle  that  changing  ESOH 
requirements  and  regulations  represent,  the  PM 
shall  regularly  review  ESOH  regulatory  require¬ 
ments  and  evaluate  their  impact  on  the  program’s 
life-cycle  cost,  schedule,  and  performance. 

Para  C5. 2.3. 5. 10.6.1.  Safety  and  Health 

The  PM  shall  identify  and  evaluate  safety  and 
health  hazards,  define  risk  levels,  and  establish 
a  program  that  manages  the  probability  and  se¬ 
verity  of  all  hazards  associated  with  develop¬ 
ment,  use,  and  disposal  of  the  system.  The  PM 
shall  use  and  require  contractors  to  use  the  in¬ 
dustry  and  DoD  standard  practice  for  system 
safety,  consistent  with  mission  requirements.  This 
standard  practice  manages  risks  encountered  in 
the  acquisition  life  cycle  of  systems,  subsystems, 
equipment,  and  facilities.  These  risks  include 
conditions  that  create  significant  risks  of  death, 
injury,  acute  or  chronic  illness,  disability,  and/or 
reduced  job  performance  of  personnel  who  pro¬ 
duce,  test,  operate,  maintain,  support,  or  dispose 
of  the  system. 

Para  C5. 2.3. 5. 10.6.2.  Safety  and  Health 

The  following  policy  applies  to  the  acceptance 
of  risk: . .  .The  PM  shall  formally  document  each 
management  decision  accepting  the  risk 
associated  with  an  identified  hazard. 

Para  C5.2.3.5.10.6.2.2.  Safety  and  Health 

“High  Risk”  hazards  shall  require  CAE  approval 
(Lead  Executive  Component  authority  prevails 
for  j  oint  programs ) . 

Para  C5.2.3.5.10.6.2.3.  Safety  and  Health 

The  acceptance  of  all  risks  involving  explosives 
safety  (see  subparagraph  C5. 2. 3. 5. 10.9.  below) 


shall  require  the  appropriate  risk  acceptance  au¬ 
thority  to  consult  with  the  DoD  Component’s 
technical  authority  managing  the  explosives 
safety  program. 

Para  C5.2.3. 5. 10.6.2.4.  Safety  and  Health 

“Serious  Risk”  hazards  shall  require  PEO 
approval. 

Para  C5. 2.3. 5. 10.6.2.5.  Safety  and  Health 

“Medium  Risk”  and  “Low  Risk”  hazards  shall 
require  PEO  approval. 

Para  C5. 2. 3. 5. 10.8.1.  Pollution  Prevention 

The  PM  shall  identify  and  evaluate  environmen¬ 
tal  and  occupational  health  hazards  and  estab¬ 
lish  a  pollution  prevention  program.  The  PM 
shall  identify  the  impacts  of  the  system  on  the 
environment  during  its  life  (including  disposal), 
the  types  and  amounts  of  pollution  from  all 
sources  (air,  water,  noise,  etc.)  that  will  be  re¬ 
leased  to  the  environment,  actions  needed  to  pre¬ 
vent  or  control  the  impacts,  ESOH  risks 
associated  with  using  the  new  system,  and  other 
information  needed  to  identify  source  reduction, 
alternative  technologies,  and  recycling 
opportunities.... 

Para  C5. 2.3. 5.13  Mission  Assuredness 

. .  .The  PM  shall  include  the  considerations  in  the 
risk  benefit  analysis  of  system  design  and  cost .... 

Para  C5.2.3. 5.15.1.  Anti-Tamper 
Provisions 

. .  .Because  of  its  function,  anti-tamper  should  not 
be  regarded  as  an  option  or  a  system  capability 
that  may  later  be  traded  off  without  a  thorough 
operational  and  acquisition  risk  analysis.  To  ac¬ 
complish  this,  the  PM  shall  identify  critical  tech¬ 
nologies,  identify  system  vulnerabilities,  and,  with 
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assistance  from  counter-intelligence  organizations, 
perform  threat  analyses  to  the  critical  technolo¬ 
gies.  The  PM  shall  research  anti- tamper  measures 
and  determine  which  best  fit  the  performance,  cost, 
schedule,  and  risk  of  the  program. 

Para  C5.3.1.  Work  Breakdown  Structure 
(WBS) 

. .  .The  PM  shall  normally  specify  contract  WBS 
elements  only  to  level  three  for  prime  contrac¬ 
tors  and  key  subcontractors.  Only  low-level 
elements  that  address  high  risk,  high  value,  or 
high  technical  interest  areas  of  a  program  shall 
require  detailed  reporting  below  level  three .... 

Para  C5. 3.2. 2. 1.5.  Implementing  a 
Performance-Based  Business  Environment 
(PBBE) 

The  PM  shall  structure  the  PBBE  to  accomplish 
the  following:  . .  .Encourage  life-cycle  risk  man¬ 
agement  versus  risk  avoidance; 

Para  C5. 3.2.2. 1.6.  Implementing  a 
Performance-Based  Business  Environment 
(PBBE) 

The  PM  shall  structure  the  PBBE  to  accomplish 
the  following:  . .  .Simplify  acquisition  and  sup¬ 
port  operating  methods  by  transferring  tasks  to 
industry  where  cost  effective,  risk-acceptable, 
commercial  capabilities  exist;  and 

Para  C6.2.2.  Intelligence  Support 

Users  shall  assess  and  evaluate  information  su¬ 
periority  requirements.  They  shall  determine  the 
vulnerability  of  IT,  including  NSS,  supporting 
infrastructures,  and  the  effectiveness  of  risk  miti¬ 
gation  methods  to  reduce  vulnerability  to  an  ac¬ 
ceptable  level. 


Para  C6.6.1.  Information  Assurance 

PMs  shall  manage  and  engineer  information  sys¬ 
tems  using  the  best  processes  and  practices 
known  to  reduce  security  risks,  including  the 
risks  to  timely  accreditation. 

Para  C6. 6.2.1.  Information  Assurance 

Accordingly,  for  each  information  system 
development,  PMs  shall:  ...Conduct  a  system 
risk  assessment  based  on  system  criticality, 
threat,  and  vulnerabilities; 

Para  C6.7.2.4.  Technology  Protection 

Technology  protection  planning  and  develop¬ 
ment  of  the  program  protection  plan  shall  begin 
early  in  the  acquisition  life  cycle.  The  following 
considerations  apply:  . .  .Security  organizations 
shall  identify  system  vulnerabilities  and  recom¬ 
mend  cost-effective  security  measures  using  risk 
management  evaluations. 

Para  C7.2.  Decision  Points 

There  are  three  types  of  decision  points:  mile¬ 
stones,  decision  reviews,  and  interim  progress 
reviews.  Each  decision  point  results  in  a  deci¬ 
sion  to  initiate,  continue,  advance,  or  terminate 
a  project  or  program  work  effort  or  phase.  The 
review  associated  with  each  decision  point  shall 
typically  address  program  progress  and  risk, 
affordability,  program  trade-offs,  acquisition 
strategy  updates,  and  the  development  of  exit 
criteria  for  the  next  phase  or  effort .... 

Para  C7.3.1.4.  Defense  Acquisition  Board 
(DAB)  Review 

The  PM  shall  brief  the  acquisition  program  to 
the  DAB  and  specifically  emphasize  technology 
maturity,  risk  management,  affordability,  criti¬ 
cal  program  information,  technology  protection, 
and  rapid  delivery  to  the  user. . . . 
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Para  C7.4.2.  Exit  Criteria 

Phase- specific  exit  criteria  normally  track 
progress  in  important  technical,  schedule,  or 
management  risk  areas .... 

Para  C7. 5.1.  Technology  Maturity 

Technology  maturity  shall  measure  the  degree 
to  which  proposed  critical  technologies  meet  pro¬ 
gram  objectives.  Technology  maturity  is  a  prin¬ 
cipal  element  of  program  risk.  A  technology 
readiness  assessment  shall  examine  program  con¬ 
cepts,  technology  requirements,  and  demon¬ 
strated  technology  capabilities  to  determine  tech¬ 
nological  maturity. 

Para  C7. 5.4.  Technology  Maturity 

TRLs  enable  consistent,  uniform,  discussions  of 
technical  maturity,  across  different  types  of  tech¬ 
nologies.  Decision  authorities  shall  consider  the 
recommended  TRLs  (or  some  equivalent  assess¬ 
ment  methodology,  e.g.,  Willoughby  templates) 
when  assessing  program  risk. . . . 

Para  C7.12.1.  Cost  Analysis  Improvement 
Group  (CAIG)  Procedures 

. .  .The  DoD  Component  responsible  for  acqui¬ 
sition  of  a  system  shall  cooperate  with  the  CAIG 
and  provide  the  cost,  programmatic,  and  techni¬ 
cal  information  required  to  estimate  costs  and 
appraise  cost  risks. . . . 

Para  C7. 1 5. 7.  Contract  Management 

Reports 

. .  .Except  for  high-cost  or  high-risk  elements, 
the  required  level  of  reporting  detail  shall  be 
limited  to  level  three  of  the  contract  WBS. 


Para  C7. 15.7. 1.2.  Contractor  Cost  Data 
Reporting  (CCDR) 

. .  .CCDR  reporting  is  not  required  for  contracts 
priced  below  $6.5  million.  The  CCDR  require¬ 
ment  on  high-risk  or  high-technical-interest  con¬ 
tracts  priced  between  $6.5  and  $42  million  is 
left  to  the  discretion  of  the  Cost  WIPT. 

Para  C7.15. 7. 1.8.1.  Level  of  Cost 
Reporting 

Routine  reporting  shall  be  at  the  contract  WBS 
level  three  for  prime  contractors  and  key  sub¬ 
contractors.  Only  low-level  elements  that  ad¬ 
dress  high-risk,  high-value,  or  high-technical- 
interest  areas  of  a  program  shall  require  detailed 
reporting  below  level  three. . . . 

4.  DoD  Directive  (DoDD)  5000.4.  OSD  Cost 
Analysis  Improvement  Group  (CAIG), 
November  24, 1992 

Para  4.1.8  Risk  Assessment 

The  CAIG  Chair  report,  in  support  of  a  mile¬ 
stone  review,  shall  include  quantitative  assess¬ 
ments  of  the  risk  in  the  estimate  of  life-cycle 
costs.  In  developing  an  assessment  of  cost  risk, 
the  CAIG  shall  consider  the  validity  of  such  pro¬ 
grammatic  assumptions  of  the  CARDs  as  EMD 
schedules,  rates  of  utilization  of  test  assets,  pro¬ 
duction  ramp  rates,  and  buy  rates,  consistent  with 
historical  information.  The  CAIG  shall  also  con¬ 
sider  uncertainties  in  inputs  to  any  cost  estimat¬ 
ing  relationships  used  in  its  estimates,  as  well  as 
the  uncertainties  inherent  in  the  calibration  of  the 
CERs,  and  shall  consider  uncertainties  in  the  fac¬ 
tors  used  in  making  any  estimates  by  analogy. 
The  CAIG  shall  consider  cost  and  schedule  risk 
implications  of  available  assessments  of  the 
program’s  technical  risks,  and  may  include  the 
results  in  its  cost-risk  assessments.  The  CAIG 
may  consider  information  on  risk  provided  by 
any  source,  although  primary  reliance  will  be 
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on  the  technical  risk  assessments  that  are  the 
responsibility  of  the  sponsoring  DoD  compo¬ 
nents,  and  of  other  OSD  offices,  in  accordance 
with  their  functional  responsibilities. 

5.  DoD  5000.4-M.  Cost  Analysis  Guidance 
and  Procedures,  December  1992 

Chapter  1: 

(Outline  of  CARD  Basic  Structure) 

Para  1.2.1.x  (..x..)  Subsystem  Description 

This  series  of  paragraphs  (repeated  for  each  sub¬ 
system)  describes  the  major  equipment  (hard¬ 
ware/software)  WBS  components  of  the  system. 
The  discussion  should  identify  which  items  are 
off-the-shelf.  The  technical  and  risk  issues  as¬ 
sociated  with  development  and  production  of  in¬ 
dividual  subsystems  also  must  be  addressed. 

Para  2.0  Technical  and  Physical 
Description 

This  section  identifies  the  program  manager’s 
assessment  of  the  program  and  the  measures 
being  taken  or  planned  to  reduce  those  risks. 
Relevant  sources  of  risk  include:  design  concept, 
technology  development,  test  requirements, 
schedule,  acquisition  strategy,  funding  availabil¬ 
ity,  contract  stability,  or  any  other  aspect  that 
might  cause  a  significant  deviation  from  the 
planned  program.  Any  related  external  technol¬ 
ogy  programs  (planned  or  on-going)  should  be 
identified,  their  potential  contribution  to  the 


program  described,  and  their  funding  prospects 
and  potential  for  success  assessed.  This  section 
should  identify  these  risks  for  each  acquisition 
phase  (DEM/VAL,  EMD,  productions  and  de¬ 
ployment,  and  O&S).  (Phase  terminology 
changed  in  DoD  5000. 2-R,  2  April  2002.) 

Chapter  2: 

(Presentation  of  Cost  Analysis  to  OSD  CAIG) 

Para  C2.2.9.  Sensitivity  Analysis 

The  sensitivity  of  projected  costs  to  critical 
program  assumptions  shall  be  examined.  Aspects 
of  the  program  to  be  subjected  to  sensitivity  analy¬ 
sis  shall  be  identified  in  the  DoD  CCA  of  program 
assumptions.  The  analysis  shall  include  factors 
such  as  learning  curve  assumptions;  technical 
risk,  i.e.,  the  risk  of  more  development  and/or 
production  effort,  changes  in  performance  char¬ 
acteristics,  schedule  alterations,  and  variations  in 
testing  requirements;  and  acquisition  strategy 
(multiyear  procurement,  dual  sourcing,  etc.). 

Para  C2.3.3  PM  Presentation 

The  Program  Manager’s  designated  represen¬ 
tative  shall  present  the  CAIG  with  the  POE  for 
each  alternative  under  construction  and  explain 
how  each  is  derived.  This  presentation  shall 
cover  the  estimates  and  estimating  procedures 
at  the  major  subcomponent  level  (e.g.,  airframe, 
engine,  major  avionics  subsystem,  etc.).  The 
presentation  should  focus  on  the  items  that  are 
cost  drivers  and/or  elements  of  high  cost  risk. 
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APPENDIX  B 


GENERIC  RISK 
MANAGEMENT  PLAN 

SAMPLE  RISK  MANAGEMENT  PLAN 


PREFACE 

DoDI  5000.2  requires  that  “PMs  and  other 
acquisition  managers  shall  continually  assess 
program  risks.”  Further,  the  Interim  Defense 
Acquisition  Guidebook  (IDAG)  states  that  for 
AC  AT  I  Programs,  “The  PM  shall  identify  the 
risk  areas  of  the  program  and  integrate  risk  man¬ 
agement  within  overall  program  management.” 
Although  the  need  for  a  risk  management  pro¬ 
gram  and  a  risk  management  process  are  ad¬ 
dressed  throughout  this  regulation,  there  is  no 
requirement  for  a  formal  Risk  Management  Plan 
(RMP).  However,  Program  Managers  (PMs) 
have  found  such  a  plan  necessary  to  focus  prop¬ 
erly  on  the  assessment  and  handling  of  program 
risk,  a  core  acquisition  management  issue  that 
Milestone  Decision  Authorities  (MDAs)  must 
rigorously  address  at  appropriate  milestones  be¬ 
fore  making  program  decisions. 

Attached  is  a  sample  format  for  a  RMP  that  is  a 
compilation  of  several  good  risk  plans  and  the 
results  of  the  DoD  Risk  Management  Working 
Group  Study.  It  represents  the  types  of 
information  and  considerations  that  a  plan, 
tailored  to  a  specific  program,  might  contain. 
There  are  also  two  examples  of  Risk  Manage¬ 
ment  Plans — one  for  an  ACAT I  or  II  Program, 
the  other  for  an  ACAT  III  or  IV  Program.  The 
Defense  Acquisition  Deskbook,  Section  2.5.2, 
has  general  guidance  and  advice  in  all  areas  of 
risk  management.  Section  2. 5. 2.4  of  the  Defense 


Acquisition  Deskbook  contains  information 
concerning  the  development  of  a  risk  manage¬ 
ment  plan.  The  information  in  this  Guide  is  con¬ 
sistent  with,  and  in  most  cases  identical  to,  the 
Defense  Acquisition  Deskbook. 

There  is  a  danger  in  providing  a  sample  docu¬ 
ment.  First  of  all,  because  it  is  written  as  a  guide 
for  a  general  audience,  it  does  not  satisfy  all  of 
the  needs  of  any  particular  program.  Second, 
there  is  the  possibility  that  some  prospective 
user  will  simply  adopt  the  plan  as  written, 
despite  the  fact  that  it  does  not  fit  his  or  her 
program.  We  discourage  this. 

The  reason  for  providing  this  sample  format  is 
to  give  PMs  and  their  staffs  a  starting  point  for 
their  own  planning  process.  It  should  stimulate 
thought  about  what  has  to  be  done  and  give 
some  ideas  on  how  to  begin  writing  a  plan.  The 
sample  plan  contains  more  information  than 
most  program  offices  should  need.  Few  PMs 
have  the  resources  for  a  dedicated  risk  man¬ 
agement  effort  as  depicted  in  the  plan.  The  key 
to  using  the  sample  plan  is  to  keep  things  simple 
and  tailor  the  plan  to  suit  your  needs,  focusing 
on  the  management  of  risk  in  the  key  critical 
areas  of  your  program. 

The  following  text  reflects  the  outline  of  a  risk 
management  plan  found  in  the  Defense 
Acquisition  Deskbook  section  2.5. 2.4. 
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SAMPLE  FORMAT  FOR 
RISK  MANAGEMENT  PLAN 


Introduction.  This  section  should  address  the 
purpose  and  objective  of  the  plan,  and  provide  a 
brief  summary  of  the  program,  to  include  the  ap¬ 
proach  being  used  to  manage  the  program,  and 
the  acquisition  strategy. 

Program  Summary.  This  section  contains  a 
brief  description  of  the  program,  including  the 
acquisition  strategy  and  the  program  manage¬ 
ment  approach.  The  acquisition  strategy  should 
address  its  linkage  to  the  risk  management 
strategy. 

Definitions.  Definitions  used  by  the  program 
office  should  be  consistent  with  DoD  defini¬ 
tions  for  ease  of  understanding  and  consistency. 
However,  the  DoD  definitions  allow  program 
managers  flexibility  in  constructing  their  risk 
management  programs.  Therefore,  each  pro¬ 
gram’s  risk  management  plan  may  include  defi¬ 
nitions  that  expand  the  DoD  definitions  to  fit 
its  particular  needs.  For  example,  each  plan 
should  include,  among  other  things,  definitions 
for  the  ratings  used  for  technical,  schedule  and 
cost  risk.  (Discussion  of  risk  rating  is  contained 
in  the  Defense  Acquisition  Deskbook  Section 
2  5  2  1  ) 

Risk  Management  Strategy  and  Approach. 

Provide  an  overview  of  the  risk  management 
approach,  to  include  the  status  of  the  risk 
management  effort  to  date,  and  a  description 
of  the  program  risk  management  strategy.  See 
the  Defense  Acquisition  Deskbook  Sections 
2.5.2. 1  and  2. 5. 2. 3. 

Organization.  Describe  the  risk  management 
organization  of  the  program  office  and  list  the 
responsibilities  of  each  of  the  risk  management 
participants.  See  the  Defense  Acquisition 
Deskbook  Section  2. 5. 2. 3. 


Risk  Management  Process  and  Procedures. 

Describe  the  program  risk  management  process 
to  be  employed;  i.e.,  risk  planning,  assessment, 
handling,  monitoring  and  documentation,  and  a 
basic  explanation  of  these  components.  See  the 
Defense  Acquisition  Deskbook  Section  2.5.2. 1. 
Also  provide  application  guidance  for  each  of 
the  risk  management  functions  in  the  process.  If 
possible,  the  guidance  should  be  as  general  as 
possible  to  allow  the  program’s  risk  management 
organization  (e.g.,  IPTs)  flexibility  in  managing 
the  program  risk,  yet  specific  enough  to  ensure 
a  common  and  coordinated  approach  to  risk 
management.  It  should  address  how  the  in-for¬ 
mation  associated  with  each  element  of  the  risk 
management  process  will  be  documented  and 
made  available  to  all  participants  in  the  pro-cess, 
and  how  risks  will  be  tracked,  to  include  the  iden¬ 
tification  of  specific  metrics  if  possible. 

Risk  Planning.  This  section  describes  the  risk 
planning  process  and  provides  guidance  on  how 
it  will  be  accomplished,  and  the  relationship  be¬ 
tween  continuous  risk  planning  and  this  RMP 
Guidance  on  updates  of  the  RMP  and  the  approval 
process  to  be  followed  should  also  be  included. 
See  Section  2.5.2. 1  of  the  Defense  Acquisition 
Deskbook  for  information  on  risk  planning. 

Risk  Assessment.  This  section  of  the  plan 
describes  the  assessment  process  and  proce¬ 
dures  for  examining  the  critical  risk  areas  and 
processes  to  identify  and  document  the  asso¬ 
ciated  risks.  It  also  summarizes  the  analyses 
process  for  each  of  the  risk  areas  leading  to 
the  determination  of  a  risk  rating.  This  rating 
is  a  reflection  of  the  potential  impact  of  the 
risk  in  terms  of  its  variance  from  known  Best 
Practices  or  probability  of  occurrence,  its 
consequence/impact,  and  its  relationship  to  other 
risk  areas  or  processes.  This  section  may  include: 
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•  Overview  and  scope  of  the  assessment 
process; 

•  Sources  of  information; 

•  Information  to  be  reported  and  formats ; 

•  Description  of  how  risk  information  is 
documented;  and 

•  Assessment  techniques  and  tools  (see 
Section  2. 5. 2. 4  of  the  Defense  Acquisition 
Deskbook). 

Risk  Handling.  This  section  describes  the 
procedures  that  can  be  used  to  determine  and 
evaluate  various  risk-handling  options,  and 
identifies  tools  that  can  assist  in  implement¬ 
ing  the  risk-handling  process.  It  also  provides 
guidance  on  the  use  of  the  various  handling 
options  for  specific  risks. 


Risk  Monitoring.  This  section  describes  the 
process  and  procedures  that  will  be  followed  to 
monitor  the  status  of  the  various  risk  events  iden¬ 
tified.  It  should  provide  criteria  for  the  selection 
of  risks  to  be  reported  on,  and  the  frequency  of 
reporting.  Guidance  on  the  selection  of  metrics 
should  also  be  included. 

Risk  Management  Information  System, 
Documentation  and  Reports.  This  section 
describes  the  MIS  structure,  rules,  and  proce¬ 
dures  that  will  be  used  to  document  the  results 
of  the  risk  management  process.  It  also  identi¬ 
fies  the  risk  management  documentation  and 
reports  that  will  be  prepared;  specifies  the  format 
and  frequency  of  the  reports;  and  assigns  respon¬ 
sibility  for  their  preparation. 
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SAMPLE  RISK  MANAGEMENT  PLAN 
FOR  THE  XYZ  PROGRAM  (ACAT  I,  II) 


1.0  INTRODUCTION 
1.1  PURPOSE 

This  Risk  Management  Plan  (RMP)  presents  the 
process  for  implementing  proactive  risk  manage¬ 
ment  as  part  of  the  overall  management  of  the 
XYZ  program.  Risk  management  is  a  program 
management  tool  to  assess  and  mitigate  events 
that  might  adversely  impact  the  program.  There¬ 
fore,  risk  management  increases  the  probability/ 
likelihood  of  program  success.  This  RMP  will: 

•  Serve  as  a  basis  for  identifying  alternatives 
to  achieve  cost,  schedule,  and  performance 
goals, 

•  Assist  in  making  decisions  on  budget  and 
funding  priorities, 

•  Provide  risk  information  for  Milestone 
decisions,  and 

•  Allow  monitoring  the  health  of  the  program 
as  it  proceeds. 

The  RMP  describes  methods  for  identifying, 
analyzing,  prioritizing,  and  tracking  risk 
drivers;  developing  risk-handling  plans;  and 
planning  for  adequate  resources  to  handle  risk. 
It  assigns  specific  responsibilities  for  the  man¬ 
agement  of  risk  and  prescribes  the  document¬ 
ing,  monitoring,  and  reporting  processes  to  be 
followed. 

This  is  the  second  edition  of  the  Risk  Manage¬ 
ment  Plan  for  the  XYZ  program.  The  initial 
plan  concentrated  on  the  tasks  within  the  Con¬ 
cept  Refinement  (CR)  and  Technology  Devel¬ 
opment  (TD)  Phases  leading  to  Milestone  B;  this 
plan  concentrates  on  the  tasks  and  activities  of 


the  System  Integration  part  of  the  System  De¬ 
velopment  and  Demonstration  (SDD)  Phase. 
Subsequent  updates  to  this  RMP  will  shift  focus 
to  the  later  acquisition  phases.  There  are 
changes  in  every  area  of  the  plan;  they  include 
refinement  of  the  risk  identification  process. 
The  PMO  Risk  Management  Coordinator  has 
been  identified  and  training  of  IPT  members  has 
commenced. 

1.2  PROGRAM  SUMMARY 

The  XYZ  program  was  initiated  in  response  to 
Mission  Need  Statement  (MNS)  XXX,  dated 
DD-MM-YYYY  and  Operational  Require¬ 
ments  Document  (ORD),  dated  DD-MM- 
YYYY.  (NOTE:  The  MNS  is  being  replaced 
by  the  Initial  Capabilities  Document.  The  ORD 
is  being  replaced  by  the  Capability  Development 
Document  (CDD)).  It  is  required  to  support  the 
fundamental  objective  of  U.S.  defense  policy  as 
stated  in  Defense  Planning  Guidance  (DPG)  and 
the  National  Military  Strategy.  The  XYZ  sys¬ 
tem  is  based  on  the  need  for  an  integrated  com¬ 
bat  system  to  link  battlefield  decision  makers. 
The  XYZ  mission  areas  are:  (Delineate  appli¬ 
cable  areas). 

The  XYZ  program  will  develop  and  procure  120 
advanced  platforms  to  replace  the  aging  ABC 
platforms  currently  in  the  inventory.  In  order 
to  meet  force  structure  objectives,  the  XYZ 
system  must  reach  Initial  Operational  Capabil¬ 
ity  (IOC)  (four  platforms)  by  FY-07.  The  pro¬ 
gram  is  commencing  an  eight-year  EMD  phase 
that  will  be  followed  by  a  five-year  procurement 
phase.  The  objectives  of  the  EMD  phase  are  to 
(discuss  the  specific  objectives  of  this  phase).  The 
program  has  Congressional  interest  and  is  re¬ 
stricted  to  a  research  and  development  funding 
ceiling  of  $300  million. 
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1.2.1  System  Description 

The  XYZ  will  be  an  affordable,  yet  capable, 
platform  taking  advantage  of  technological 
simplification  and  advancements.  The  XYZ 
integrated  Combat  System  includes  all  non¬ 
propulsion  electronics  and  weapons.  Sub¬ 
systems  provide  capabilities  in  combat  control, 
electronic  warfare  support  measures  (ESM), 
defensive  warfare,  navigation,  radar,  interior 
communications,  monitoring,  data  transfer, 
tactical  support  device,  exterior  communica¬ 
tions,  and  Identification  Friend  or  Foe  (IFF). 
Weapons  systems  are  to  be  provided  by  the 
program  offices  that  are  responsible  for  their 
development.  The  Mechanical  and  Electrical 
(M&E)  system  comprises....  The  Combat  Sys¬ 
tem,  M&E  systems,  and  subsystems  provide  the 
XYZ  system  with  the  capability  and  connec¬ 
tivity  to  accomplish  the  broad  range  of  missions 
defined  in  the  MNS  and  ORD. 

1.2.2  Acquisition  Strategy 

The  XYZ  program  initial  strategy  is  to  contract 
with  one  prime  contractor  in  the  System 
Integration  part  of  the  System  Development  and 
Demonstration  Phase  for  development  of  two 
prototype  systems  for  test  and  design  valida¬ 
tion.  Due  to  the  technical  complexity  of  achiev¬ 
ing  the  performance  levels  of  the  power  gen¬ 
eration  systems,  the  prime  will  use  two  sub¬ 
contractors  for  the  engine  development  and 
down  select  to  one  producer  prior  to  low  rate 
initial  production,  which  is  scheduled  for  FY- 
04.  Various  organizations,  such  as  the  Govern¬ 
ment  Research  Faboratory  will  be  funded  to  pro¬ 
vide  experts  for  assessment  of  specific  areas  of 
risk.  The  program  has  exit  criteria,  included  in 
the  list  of  Critical  Program  Attributes  in  Annex 
A,  that  must  be  met  before  progressing  to  the 
next  phase. 


1.2.3  Program  Management  Approach 

The  XYZ  program  is  managed  using  the  IPPD 
concept,  with  program  integrated  product  teams 
(PIPTs)  established  largely  along  the  hierarchy 
of  the  product  work  breakdown  structure 
(WBS).  There  are  also  cost-performance  and 
test  Working  IPTs  (WIPTs)  established  for  ver¬ 
tical  coordination  up  the  chain  of  command. 
The  PM  chairs  a  program  level  IPT  (PFIPT) 
that  addresses  issues  that  are  not  resolved  at 
the  WIPT  or  PIPT  level. 

1.3  DEFINITIONS 

1.3.1  Risk 

Risk  is  a  measure  of  the  inability  to  achieve 
overall  program  objectives  within  defined  cost, 
schedule,  and  technical  constraints  and  has  two 
components:  (1)  the  probability  of  failing  to 
achieve  a  particular  outcome  and  (2)  the 
consequences/impacts  of  failing  to  achieve  that 
outcome.  For  processes,  risk  is  a  measure  of 
the  difference  between  actual  performance  of 
a  process  and  the  known  best  practice  for 
performing  that  process. 

1.3.2  Risk  Event 

Risk  events  are  those  events  within  the  XYZ 
program  that,  if  they  go  wrong,  could  result  in 
problems  in  the  development,  production,  and 
fielding  of  the  system.  Risk  events  should  be 
defined  to  a  level  such  that  the  risk  and  causes 
are  understandable  and  can  be  accurately  as¬ 
sessed  in  terms  of  probability/likelihood  and 
consequence/impact  to  establish  the  level  of 
risk.  For  processes,  risk  events  are  assessed  in 
terms  of  process  variance  from  known  best 
practices  and  potential  consequences/impacts 
of  the  variance. 
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1.3.3  Technical  Risk 

This  is  the  risk  associated  with  the  evolution  of 
the  design  and  the  production  of  the  XYZ  system 
affecting  the  level  of  performance  necessary  to 
meet  the  operational  requirements.  The 
contractor’s  and  subcontractors’  design,  test,  and 
production  processes  (process  risk)  influence  the 
technical  risk  and  the  nature  of  the  product  as 
depicted  in  the  various  levels  of  the  Work  Break¬ 
down  Structure  (product  risk). 

1.3.4  Cost  Risk 

This  is  the  risk  associated  with  the  ability  of 
the  program  to  achieve  its  life-cycle  cost 
objectives.  Two  risk  areas  bearing  on  cost  are 
(1)  the  risk  that  the  cost  estimates  and  objec¬ 
tives  are  accurate  and  reasonable  and  (2)  the 
risk  that  program  execution  will  not  meet  the 
cost  objectives  as  a  result  of  a  failure  to  handle 
cost,  schedule,  and  performance  risks. 

1.3.5  Schedule  Risk 

These  risks  are  those  associated  with  the  ad¬ 
equacy  of  the  time  estimated  and  allocated  for 
the  development,  production,  and  fielding  of  the 
system.  Two  risk  areas  bearing  on  schedule  risk 
are  (1)  the  risk  that  the  schedule  estimates  and 
objectives  are  realistic  and  reasonable  and  (2) 
the  risk  that  program  execution  will  fall  short 
of  the  schedule  objectives  as  a  result  of  failure 
to  handle  cost,  schedule,  or  performance  risks. 

1.3.6  Risk  Ratings 

This  is  the  value  that  is  given  to  a  risk  event  (or 
the  program  overall)  based  on  the  analysis  of  the 
probability/likelihood  and  consequences/impacts 
of  the  event.  For  the  XYZ  program,  risk  ratings 
of  Low,  Moderate,  or  High  will  be  assigned  based 
on  the  following  criteria.  See  Section  3.3.2  of  this 
appendix  for  guidance  on  determining  probabil¬ 
ity/likelihood  and  consequences/impacts.  When 


rating  process  variance  from  best  practices,  there 
is  no  rating  of  probability/likelihood,  rather  the 
level  would  be  a  measure  of  the  variance  from 
best  practices  (see  Paragraph  3. 3. 2. 3). 

•  Low  Risk:  Has  little  or  no  potential  for 
increase  in  cost,  disruption  of  schedule,  or 
degradation  of  performance.  Actions  within 
the  scope  of  the  planned  program  and  nor¬ 
mal  management  attention  should  result  in 
controlling  acceptable  risk. 

•  Moderate  Risk:  May  cause  some  increase 
in  cost,  disruption  of  schedule,  or  degrada¬ 
tion  of  performance.  Special  action  and  man¬ 
agement  attention  may  be  required  to  handle 
risk. 

•  High  Risk:  Likely  to  cause  significant 
increase  in  cost,  disruption  of  schedule,  or 
degradation  of  performance.  Significant 
additional  action  and  high  priority  manage¬ 
ment  attention  will  be  required  to  handle  risk. 

1.3.7  Independent  Risk  Assessor 

An  independent  risk  assessor  is  a  person  who  is 
not  in  the  management  chain  or  directly  involved 
in  performing  the  tasks  being  assessed.  Use  of 
independent  risk  assessors  is  a  valid  technique 
to  ensure  that  all  risk  areas  are  identified  and  that 
the  consequence/impact  and  probability /likeli¬ 
hood  (or  process  variance)  are  properly  under¬ 
stood.  The  technique  can  be  used  at  different 
program  levels,  e.g.,  Program  Office,  Service 
Field  Activities,  Contractors,  etc.  The  Program 
Manager  will  approve  the  use  of  independent 
assessors,  as  needed. 

1.3.8  Templates  and  Best  Practices 

A  “template”  is  a  disciplined  approach  for  the  ap¬ 
plication  of  critical  engineering  and  manufactur¬ 
ing  processes  that  are  essential  to  the  success  of 
most  programs.  DoD  4245. 7-M,  Transition  from 
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Development  to  Production  Solving  the  Risk 
Equation ,  provides  a  number  of  such  templates. 
For  each  template  process  described  in  DoD 
4245. 7-M,  Best  Practice  Information  is  described 
in  NAVSO  P-607 1 .  These  documents  outline  the 
ideal  or  low  risk  approach  and  thus  serve  as  a 
baseline  from  which  risk  for  some  XYZ  processes 
can  be  assessed. 

1.3.9  Metrics 

There  are  measures  used  to  indicate  progress 
or  achievement. 

1.3.10  Critical  Program  Attributes 

Critical  Program  Attributes  are  performance, 
cost,  and  schedule  properties  or  values  that  are 
vital  to  the  success  of  the  program.  They  are  de¬ 
rived  from  various  sources,  such  as  the 
Acquisition  Program  Baseline,  exit  criteria  for 
the  next  program  phase,  Key  Performance 
Parameters,  test  plans,  the  judgment  of  program 


experts,  etc.  The  XYZ  program  will  track  these 
attributes  to  determine  the  progress  in  achieving 
the  final  required  value.  See  Annex  A  for  a  list 
of  the  XYZ  Critical  Program  Attributes. 

2.0  RISK  MANAGEMENT  APPROACH 

2.1  GENERAL  APPROACH  AND 
STATUS 

DoDI  5000.2  states:  “Risks  must  be  well  under¬ 
stood,  and  risk  management  approaches  devel¬ 
oped,  before  decision  authorities  can  authorize 
a  program  to  proceed  into  the  next  phase  of  the 
acquisition  process.”  This  policy  is  implemented 
in  DoD  Regulation  5000. 2-R,  with  more  de¬ 
tailed  guidance  provided  in  the  individual  Ser¬ 
vice  regulation.  The  Defense  Acquisition  Desk- 
book  (Section  2.5.2)  provides  additional  guid¬ 
ance,  advice,  and  wisdom  on  the  management 
of  risk.  Figure  B-l  shows  how  the  XYZ  pro¬ 
gram  risk  management  fits  into  the  phases  and 
milestones  of  the  acquisition  process. 


Current  Status 

•  Baseline 

-  Cost 

-  Schedule 

-  Performance 

•  Execution  Status 

Current  Status 

•  Refined  Baseline 

-  Cost 

-  Schedule 

-  Performance 

•  Execution  Status 

Plans 

Risk 

Plans 

•  Program  Plans 

Management 

•  Program  Plans 

•  Exit  Criteria 

/ 

•  Exit  Criteria 

Assessment 

Assessment 

•  Cost 

•  Cost 

•  Schedule 

•  Schedule 

•  Performance 

•  Performance 

Figure  B-1 .  Risk  Management  and  the  Acquisition  Process 


B-7 


The  XYZ  program  will  use  a  centrally  devel¬ 
oped  risk  management  strategy  throughout  the 
acquisition  process  and  decentralized  risk  plan¬ 
ning,  assessment,  handling,  and  monitoring. 
XYZ  risk  management  is  applicable  to  all 
acquisition  functional  areas. 

The  results  of  the  Concept  Exploration  Phase 
of  the  program  identified  potential  risk  events 
and  the  Acquisition  Strategy  reflects  the 
program’s  risk-handling  approach.  Overall,  the 
risk  of  the  XYZ  program  for  Milestone  B  was 
assessed  as  moderate,  but  acceptable.  Moder¬ 
ate  risk  functional  areas  were  threat,  manufac¬ 
turing,  cost,  funding,  and  schedule.  The  remain¬ 
ing  functional  areas  of  technology,  design  and 
engineering  (hardware  and  software),  support, 
(schedule)  concurrency,  human  systems 
integration,  and  environmental  impact  were 
assessed  as  low  risk. 

2.2  RISK  MANAGEMENT  STRATEGY 

The  basic  risk  management  strategy  is  intended 
to  identify  critical  areas  and  risk  events,  both  tech¬ 
nical  and  non-technical,  and  take  necessary  ac¬ 
tion  to  handle  them  before  they  can  become 
problems,  causing  serious  cost,  schedule,  or  per¬ 
formance  impacts.  This  program  will  make  ex¬ 
tensive  use  of  modeling  and  simulation,  tech¬ 
nology  demonstrations,  and  prototype  testing 
in  handling  risk. 

Risk  management  will  be  accomplished  using 
the  integrated  Government-Contractor  IPT  or¬ 
ganization.  These  IPTs  will  use  a  structured  as¬ 
sessment  approach  to  identify  and  analyze  those 
processes  and  products  that  are  critical  to  meet¬ 
ing  the  program  objectives.  They  will  then  de¬ 
velop  risk-handling  options  to  mitigate  the  risks 
and  monitor  the  effectiveness  of  the  selected 
handling  options.  Key  to  the  success  of  the  risk 
management  effort  is  the  identification  of  the 
resources  required  to  implement  the  developed 
risk-handling  options. 


Risk  information  will  be  captured  by  the  IPTs  in 
a  risk  management  information  system  (RMIS) 
using  a  standard  Risk  Information  Form  (RIF). 
The  RMIS  will  provide  standard  reports,  and  is 
capable  of  preparing  ad  hoc  tailored  reports.  See 
Annex  B  for  a  description  of  the  RMIS  and  RIF. 

Risk  information  will  be  included  in  all  program 
reviews,  and  as  new  information  becomes  avail¬ 
able,  the  PMO  and  contractor  will  conduct  ad¬ 
ditional  reviews  to  ascertain  if  new  risks  exist. 
The  goal  is  to  be  continuously  looking  to  the 
future  for  areas  that  may  severely  impact  the 
program. 

2.3  ORGANIZATION 

The  risk  organization  for  the  XYZ  program  is 
shown  in  Figure  B-2.  This  is  not  a  separate 
organization,  but  rather  shows  how  risk  is 
integrated  into  the  program’s  existing  organiza¬ 
tion  and  shows  risk  relationships  among  mem¬ 
bers  of  the  program  team. 

2.3.1  Risk  Management  Coordinator 

The  Risk  Management  Coordinator,  the  XYZ 
Technology  Assessment  and  R&D  Manager,  is 
overall  coordinator  of  the  Risk  Management 
Program.  The  Risk  Management  Coordinator  is 
responsible  for: 

•  Maintaining  this  Risk  Management  Plan; 

•  Maintaining  the  Risk  Management  Database; 

•  Briefing  the  PM  on  the  status  of  XYZ 
program  risk; 

•  Tracking  efforts  to  reduce  moderate  and  high 
risk  to  acceptable  levels; 

•  Providing  risk  management  training ; 

•  Facilitating  risk  assessments;  and 
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Figure  B-2.  XYZ  Risk  Management  Organization 


•  Preparing  risk  briefings,  reports,  and  docu¬ 
ments  required  for  Program  Reviews  and  the 
acquisition  Milestone  decision  processes. 

2.3.2  Program  Level  Integrated  Product 
Team  (PLIPT) 

The  PLIPT  is  responsible  for  complying  with 
the  DoD  risk  management  policy  and  for  struc¬ 
turing  an  efficient  and  useful  XYZ  risk  manage¬ 
ment  approach.  The  Program  Manager  is  the 
Chair  of  the  PLIPT.  The  PLIPT  membership 
may  be  adjusted  but  is  initially  established  as  the 
chairs  of  the  Program  IPTs,  designated  sub-tier 
IPTs,  and  the  Heads  of  PMO  Functional  Offices. 

2.3.3  PIPTs 

The  Program  IPTs  are  responsible  for  imple¬ 
menting  risk  management  tasks  per  this  plan. 
This  includes  the  following  responsibilities: 


•  Quarterly,  or  as  directed,  update  the  program 
risk  assessments  made  during  the  System 
Integration  (SI)  part  of  the  System  Develop¬ 
ment  and  Demonstration  (SDD)  Phase. 

•  Review  and  be  prepared  to  justify  the  risk 
assessments  made  and  the  risk  handling  plan 
proposed. 

•  Report  risk  to  the  Program  Manager/Program 
Director,  with  information  to  the  Risk  Man¬ 
agement  Coordinator  via  Risk  Information 
Forms  (RIFs). 

•  Ensure  that  risk  is  a  consideration  at  each  Pro¬ 
gram  and  Design  Review. 

•  Ensure  Design/Build  Team  responsibilities 
incorporate  appropriate  risk  management 
tasks. 

2.3.4  XYZ  Independent  Risk  Assessors 


Review  and  recommend  to  the  Risk  Manage¬ 
ment  Coordinator  changes  on  the  overall  risk 
management  approach  based  on  lessons  learned. 
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Independent  Assessors  made  a  significant 
contribution  to  the  XYZ  Milestone  B  risk 


assessments.  The  use  of  independent  assess¬ 
ments  as  a  means  of  ensuring  that  all  risk  areas 
are  identified  will  continue,  when  necessary. 

2.3.5  Other  Risk  Assessment 
Responsibilities 

The  Risk  Assessment  responsibilities  of  other 
Systems  Command  codes,  Service  Field  Activi¬ 
ties,  Design/Build  Teams,  and  Contractors  will 
be  as  described  in  Memoranda  of  Agreement 
(MOAs),  Memoranda  of  Understanding 
(MOUs),  Systems  Command  Tasking,  or  con¬ 
tracts.  This  RMP  should  be  used  as  a  guide  for 
XYZ  risk  management  efforts. 

2.3.6  User  Participation 

The  Requirements  Organization  (specific  code) 
is  the  focal  point  for  providing  the  Program 
Executive  Officer  or  the  Project  Manager  with 
user  identified  risk  assessments. 

2.3.7  Risk  Training 

The  key  to  the  success  of  the  risk  efforts  is  the 
degree  to  which  all  members  of  the  team,  both 
Government  and  contractor  are  properly  trained. 
The  XYZ  Program  Office  will  provide  risk  train¬ 
ing,  or  assign  members  to  training  classes,  dur¬ 
ing  the  SDD  Phase.  Key  personnel  with  XYZ 


management  or  assessment  responsibilities  are 
required  to  attend.  All  members  of  the  team  will 
receive,  at  a  minimum,  basic  risk  man-agement 
training.  XYZ  sponsored  training  is  planned  to 
be  presented  according  to  the  schedule  provided 
in  Annex  X  (not  provided). 

3.0  RISK  MANAGEMENT  PROCESS 
AND  PROCEDURES 

3.1  OVERVIEW 

This  section  describes  XYZ  program’s  risk 
management  process  and  provides  an  overview 
of  the  XYZ  risk  management  approach.  The 
Defense  Acquisition  Deskbook  defines  risk 
management  as  “the  act  or  practice  of  control¬ 
ling  risk.  It  includes  risk  planning,  assessing 
risk  areas,  developing  risk-handling  options, 
monitoring  risks  to  determine  how  risks  have 
changed,  and  documenting  the  overall  risk 
management  program.”  Figure  B-3  shows,  in 
general  terms,  the  overall  risk  management  pro¬ 
cess  that  will  be  followed  in  the  XYZ  program. 
This  process  follows  DoD  and  Service  policies 
and  guidelines  and  incorporates  ideas  found  in 
other  sources.  Each  of  the  risk  management 
functions  shown  in  Figure  B-3  is  discussed  in 
the  following  paragraphs,  along  with  specific  pro¬ 
cedures  for  executing  them. 
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Figure  B-3.  Risk  Management  Structure 
(also  referred  to  as  the  Risk  Management  Process  Model) 
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3.2  RISK  PLANNING 

3.2.1  Process 

Risk  planning  consists  of  the  up-front  activities 
necessary  to  execute  a  successful  risk  manage¬ 
ment  program.  It  is  an  integral  part  of  normal 
program  planning  and  management.  The  plan¬ 
ning  should  address  each  of  the  other  risk  man¬ 
agement  functions,  resulting  in  an  organized  and 
thorough  approach  to  assess,  handle,  and  moni¬ 
tor  risks.  It  should  also  assign  responsibilities  for 
specific  risk  management  actions  and  establish 
risk  reporting  and  documentation  requirements. 
This  RMP  serves  as  the  basis  for  all  detailed  risk 
planning,  which  must  be  continuous. 

3.2.2  Procedures 

3.2.2. 1  Responsibilities.  Each  IPT  is  respon¬ 
sible  for  conducting  risk  planning,  using  this 
RMP  as  the  basis.  The  planning  will  cover  all 
aspects  of  risk  management  to  include  assess¬ 
ment,  handling  options,  and  monitoring  of  risk 
handling  activities.  The  Program  Risk  Manage¬ 
ment  Coordinator  will  monitor  the  planning 
activities  of  the  IPTs  to  ensure  that  they  are  con¬ 
sistent  with  this  RMP  and  that  appropriate  revi¬ 
sions  to  this  plan  are  made  when  required  to  re¬ 
flect  significant  changes  resulting  from  the  IPT 
planning  efforts. 

Each  person  involved  in  the  design,  production, 
operation,  support,  and  eventual  disposal  of  the 
XYZ  system  or  any  of  its  systems  or  compo¬ 
nents  is  a  part  of  the  risk  management  process. 
This  involvement  is  continuous  and  should  be 
considered  a  part  of  the  normal  management 
process. 

3.2.2.2  Resources  and  Training.  An  effective 
risk  management  program  requires  resources. 
As  part  of  its  planning  process,  each  IPT  will 
identify  the  resources  required  to  implement  the 
risk  management  actions.  These  resources 


include  time,  material,  personnel,  and  cost.  Train¬ 
ing  is  major  consideration.  All  IPT  members 
should  receive  instruction  on  the  fundamentals 
of  risk  management  and  special  training  in  their 
area  of  responsibility,  if  necessary. 

3.2.2.3  Documentation  and  Reporting.  This 
RMP  establishes  the  basic  documentation  and  re¬ 
porting  requirements  for  the  program.  IPTs  should 
identify  any  additional  requirements  that  might 
be  needed  to  effectively  manage  risk  at  their  level. 
Any  such  additional  requirements  must  not  con¬ 
flict  with  the  basic  requirements  in  this  RMP. 

3.2.2.4  Metrics.  Each  IPT  should  establish 
metrics  that  will  measure  the  effectiveness  of 
their  planned  risk-handling  options.  See  Annex 
C  for  an  example  of  metrics  that  may  be  used. 

3.2.2.5  Risk  Planning  Tools.  The  following 
tools  can  be  useful  in  risk  planning.  It  may  be 
useful  to  provide  this  information  to  the  contrac¬ 
tors  to  help  them  understand  the  XYZ  program’s 
approach  to  managing  risk.  This  list  is  not  meant 
to  be  exclusive. 

•  DoD  Manual  4245 .7-M,  a  DoD  guide  for 
assessing  process  technical  risk. 

•  The  Navy’s  Best  Practices  Manual,  NAVSO 
P-6071,  provides  additional  insight  into  each 
of  the  Templates  in  DoD  4245 .7-M  and  a 
checklist  for  each  template. 

•  Program  Manager’s  Work  Station  (PMWS) 
software,  may  be  useful  to  some  risk  assessors. 
PMWS  has  a  Risk  Assessment  module  based 
on  the  Template  Manual  and  Best  Practices 
Manual. 

•  Commercial  and  Government  developed  risk 
management  software. 

The  latter  includes  Government  software,  such 
as  Risk  Matrix  developed  by  Mitre  Corporation 
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for  the  Air  Force  and  the  New  Attack  Subma¬ 
rine  Program’s  On-Line  Risk  Data  Base 
(OLRDB). 

3.2.2.6  Plan  Update.  This  RMP  will  be  up¬ 
dated,  if  necessary,  on  the  following  occasions: 
(1)  whenever  the  acquisition  strategy  changes, 
or  there  is  a  major  change  in  program  empha¬ 
sis;  (2)  in  preparation  for  major  decision  points; 
(3)  in  preparation  for  and  immediately  follow¬ 
ing  technical  audits  and  reviews;  (4)  concur¬ 
rent  with  the  review  and  update  of  other 
program  plans;  and  (5)  in  preparation  for  a  POM 
submission. 

3.3  RISK  ASSESSMENT 

The  risk  assessment  process  includes  the 
identification  of  critical  risk  events/processes, 
which  could  have  an  adverse  impact  on  the 
program,  and  the  analyses  of  these  events/ 
processes  to  determine  the  probability/likeli¬ 
hood  of  occurrence/process  variance  and 
consequences/impacts.  It  is  the  most  demand¬ 
ing  and  time-consuming  activity  in  the  risk 
management  process. 

3.3.1  Process 

3.3. 1. 1  Identification.  Risk  identification  is  the 
first  step  in  the  assessment  process.  The  basic  pro¬ 
cess  involves  searching  through  the  entire  XYZ 
program  to  determine  those  critical  events  that 
would  prevent  the  program  from  achieving  its 
objectives.  All  identified  risks  will  be  documented 
in  the  RMIS,  with  a  statement  of  the  risk  and  a 
description  of  the  conditions  or  situations  caus¬ 
ing  concern  and  the  context  of  the  risk. 

Risks  will  be  identified  by  all  Program  IPTs  and 
by  any  individual  in  the  program.  The  lower- 
level  IPTs  can  identify  significant  concerns  ear¬ 
lier  than  otherwise  might  be  the  case  and  iden¬ 
tify  those  events  in  critical  areas  that  must  be 
dealt  with  to  avoid  adverse  consequences/ 


impacts.  Likewise,  individuals  involved  in  the 
detailed  and  day-to-day  technical,  cost,  and 
scheduling  aspects  of  the  program  are  most 
aware  of  the  potential  problems  (risks)  that  need 
to  be  managed. 

3.3.1.2  Analysis.  This  process  involves: 

•  Identification  of  WBS  elements 

•  Evaluation  of  the  WBS  elements  using  the 
risk  areas  to  determine  risk  events 

•  As  signment  of  probability/likelihood  and  con¬ 
sequence/impact  to  each  risk  event  to  establish 
a  risk  rating 

•  Prioritization  of  each  risk  event  relative  to 
other  risks. 

Risk  analysis  should  be  supported  by  a  study, 
test  results,  modeling  and  simulation,  trade 
study,  the  opinion  of  a  qualified  expert  (to 
include  justification  of  his  or  her  judgment),  or 
any  other  accepted  analysis  technique.  The  De¬ 
fense  Acquisition  Deskbook,  Section  2524.2  de¬ 
scribes  a  number  of  analysis  techniques  that  may 
be  useful.  Evaluators  should  identify  all  assump¬ 
tions  made  in  assessing  risk.  When  appropri¬ 
ate,  a  sensitivity  analysis  should  be  done  on 
assumptions. 

Systems  engineering  analysis,  risk  assessments, 
and  manpower  risk  assessments  provide  addi¬ 
tional  information  that  must  be  considered.  This 
includes,  among  other  things,  environmental  im¬ 
pact,  system  safety  and  health  analysis,  and  se¬ 
curity  considerations.  Classified  programs 
may  experience  difficulties  in  access,  facilities, 
and  visitor  control  that  can  introduce  risk  and 
must  be  considered. 

The  analysis  of  individual  risk  will  be  the 
responsibility  of  the  IPT  identifying  the  risk,  or 
the  IPT  to  which  the  risk  has  been  assigned.  They 
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may  use  external  resources  for  assistance,  such 
as  field  activities,  Service  laboratories,  and  con¬ 
tractors.  The  results  of  the  analysis  of  all  identi¬ 
fied  risks  must  be  documented  in  the  RMIS. 

3.3.2  Procedures 

3.3.2. 1  Assessments  -  General.  Risk  assess¬ 
ment  is  an  iterative  process,  with  each  assess¬ 
ment  building  on  the  results  of  previous  assess¬ 
ments.  The  current  baseline  assessment  is  a 
combination  of  the  risk  assessment  delivered  by 
the  contractors  as  part  of  the  Concept  Refine¬ 
ment  (CR)  Phase,  the  program  office  risk  assess¬ 
ment  done  before  Milestone  B,  and  the  post¬ 
award  Integrated  Baseline  Review  (IBR)  per¬ 
formed  in  the  SI  part  of  SDD. 

For  the  program  office,  unless  otherwise  directed 
in  individual  tasking,  program  level  risk  assess¬ 
ments  will  be  presented  at  each  Program  Review 
meeting  with  a  final  update  not  later  than  6 
months  before  the  next  scheduled  Milestone 
decision.  The  primary  source  of  information  for 
the  next  assessment  will  be  the  current  assess¬ 
ment  baseline,  and  existing  documentation  such 
as  CR  Phase  study  results,  the  design  mission 
profile  that  was  done  as  part  of  the  CR  Phase, 
the  IBR,  which  will  be  conducted  immediately 
after  the  System  Integration  (SI)  Part  of  the  Sys¬ 
tem  Development  and  Demonstration  (SDD) 
Phase  contract  award,  the  contract  WBS  that  is 
part  of  the  IBR,  industry  best  practices  as  de¬ 
scribed  in  the  PMWS  Knowledge  base,  the 
ORD,  the  Acquisition  Program  Baseline  (APB), 
and  any  contractor  design  documents. 

IPTs  should  continually  assess  the  risks  in  their 
areas,  reviewing  risk-handling  actions  and  the 
critical  risk  areas  whenever  necessary  to  assess 
progress.  For  contractors,  risk  assessment 
updates  should  be  made  as  necessary. 

The  risk  assessment  process  is  intended  to  be 
flexible  enough  so  that  field  activities,  service 


laboratories,  and  contractors  may  use  their  judg¬ 
ment  in  structuring  procedures  considered  most 
successful  in  identifying  and  analyzing  all  risk 
areas. 

3.3.2.2  Identification.  Following  is  a  descrip¬ 
tion  of  step-by-step  procedures  that  evaluators 
may  use  as  a  guide  to  identify  program  risks. 

•  Step  One  -  Understand  the  requirements  and 
the  program  performance  goals,  which  are 
defined  as  thresholds  and  objectives  (see 
5000. 2-R).  Describe  the  operational  (func¬ 
tional  and  environmental)  conditions  under 
which  the  values  must  be  achieved  by 
referring  or  relating  to  design  documents.  The 
ORD  and  APB  contain  Key  Performance  Pa¬ 
rameters  (KPPs). 

•  Step  Two  -  Determine  the  engineering  and 
manufacturing  processes  that  are  needed  to 
design,  develop,  produce,  and  support  the 
system.  Obtain  industry  best  practices  for 
these  processes. 

•  Step  Three  -  Identify  contract  WBS  elements 
(to  include  products  and  processes). 

•  Step  Four  -  Evaluate  each  WBS  element 
against  sources/areas  of  risk  described  in 
Table  4-2  of  the  DSMC  Risk  Management 
Guide ,  plus  other  sources/areas  as 
appropriate. 

•  Step  Five  -  Assign  a  probability  and  conse¬ 
quence/impact  to  each  risk  event 

•  Step  Six  -  Prioritize  the  risk  events . 

Following  are  indicators  that  IPTs  may  find 
helpful  in  identifying  and  assessing  risk: 

•  Lack  of  Stability,  Clarity,  or  Understand¬ 
ing  of  Requirements:  Requirements  drive  the 
design  of  the  system.  Changing  or  poorly 
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stated  requirements  guarantees  the  intro¬ 
duction  of  performance,  cost,  and  schedule 
problems. 

•  Failure  to  Use  Best  Practices  virtually  as¬ 
sures  that  the  program  will  experience  some 
risk.  The  further  a  contractor  deviates  from 
best  practices,  the  higher  the  risk. 

•  New  Processes  should  always  be  suspect, 
whether  they  are  related  to  design,  analysis, 
or  production.  Until  they  are  validated,  and 
until  the  people  who  implement  them  have 
been  trained  and  have  experience  in  success¬ 
fully  using  the  process,  there  is  risk. 

•  Any  Process  Lacking  Rigor  should  also  be 
suspect;  it  is  inherently  risky.  To  have  rigor, 
a  process  should  be  mature  and  documented, 
it  should  have  been  validated,  and  it  should 
be  strictly  followed. 

•  Insufficient  Resources:  People,  funds, 
schedule,  and  tools  are  necessary  ingredi¬ 
ents  for  successfully  implementing  a  pro¬ 
cess.  If  any  are  inadequate,  to  include  the 
qualifications  of  the  people,  there  is  risk. 

•  Test  Failure  may  indicate  corrective  action 
is  necessary.  Some  corrective  actions  may  not 
fit  available  resources,  or  the  schedule,  and 
(for  other  reasons  as  well)  may  contain  risk. 

•  Qualified  Supplier  Availability:  A  supplier 
not  experienced  with  the  processes  for  design¬ 
ing  and  producing  a  specific  product  is  not  a 
qualified  supplier  and  is  a  source  of  risk. 

•  Ne gative  Trends  or  F orecasts  are  cause  for 
concern  (risk)  and  may  require  specific 
actions  to  turn  around. 

There  are  a  number  of  techniques  and  tools  avail¬ 
able  for  identifying  risks.  Among  them  are: 


•  Best  Judgment:  The  knowledge  and  experi¬ 
ence  of  the  collective,  multi-disciplined 
Integrated  Project  Team  (IPT)  members  and  the 
opinion  of  subject-matter  experts  (SMEs)  are 
the  most  common  source  of  risk  identification. 

•  Lessons  Learned  from  similar  processes  can 
serve  as  a  baseline  for  the  successful  way  to 
achieve  requirements.  If  there  is  a  departure 
from  the  successful  way,  there  may  be  risk. 

•  DoD  4245.7-M,  Transition  from  Develop¬ 
ment  to  Production,  is  often  called  the  “Tem¬ 
plates”  book  because  it  identifies  technical 
risk  areas  and  provides,  in  “bullet”  form,  sug¬ 
gestions  for  avoiding  those  risks.  It  focuses 
on  the  technical  details  of  product  design,  test, 
and  production  to  help  managers  proactively 
manage  risk.  It  also  includes  chapters  on  fa¬ 
cilities,  logistics,  and  management,  which 
make  this  a  useful  tool  in  identifying  weak 
areas  of  XYZ  planned  processes  early  enough 
to  implement  actions  needed  to  avoid  adverse 
consequences/impacts.  A  copy  of  this  manual 
is  available  at:  http://www.dtic.mil/whs/ 
directives. 

•  The  NAYSO  P-6071  Best  Practices  Man¬ 
ual  was  developed  by  the  Navy  to  add  depth 
to  the  Template  Book,  DoD  4245.7-M. 

•  Critical  Program  Attributes  are  metric  s  that 
the  program  office  developed  to  measure 
progress  toward  meeting  our  objectives.  Team 
members,  IPTs,  functional  managers,  contrac¬ 
tors,  etc.,  may  develop  their  own  metrics  to 
support  these  measurements.  The  attributes 
may  be  specification  requirements,  contract 
requirements,  or  measurable  parameters  from 
any  agreement  or  tasking.  The  idea  is  to  pro¬ 
vide  a  means  to  measure  whether  we  are  on 
track  in  achieving  our  objectives. 

•  Methods  and  Metrics  for  Product  Success 

is  a  manual  published  by  the  Office  of  the 
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Assistant  Secretary  of  the  Navy  (RDA)  Prod¬ 
uct  Integrity  Directorate.  It  highlights  areas 
related  to  design,  test,  and  production  pro¬ 
cesses  where  problems  are  most  often  found 
and  metrics  for  the  measurement  of  effec¬ 
tiveness  of  the  processes.  It  also  describes 
the  software  tool,  Program  Manager’s  Work 
Station  (PMWS).  (See  next  paragraph.) 

•  PMWS  contains  risk  management  software, 
“Technical  Risk  Identification  and  Mitigation 
System  (TRIMS)  and  Knowledgebase.”  They 
provide  a  tailorable  management  system 
based  on  NAVSO  P-6071  and  DoD  4245.7- 
M.  The  PMWS  provides  a  compact  disk 
(CD)  that  contains  the  necessary  programs  for 
assessing  a  program’s  risk  and  software  for 
program  management.  PMWS  can  be  ob¬ 
tained  by  calling  the  Best  Manufacturing  Pro¬ 
gram  (BMP)  Office  at  (301)  403-8100. 

•  New  Nuclear  Submarine  (NSSN)  On-Line 
Risk  Database  (ONLRB  )  is  a  software  tool 
may  be  used  to  support  the  XYZ  Risk  Man¬ 
agement  Process.  The  tool  helps  IPTs  in  the 
identification  and  assessment  of  risk  and 
management  of  handling  efforts. 

•  Risk  Matrix  is  another  candidate  for  use  by 
the  PMO.  It  is  an  automated  tool,  developed 
by  Mitre  Corporation,  that  supports  a 
structured  approach  for  identifying  risk  and 
assessing  its  potential  program  impact.  It  is 
especially  helpful  for  prioritizing  risks. 

•  Requirements  Documents  describe  the 
output  of  our  efforts.  IPT  efforts  need  to  be 
monitored  continuously  to  ensure  require¬ 
ments  are  met  on  time  and  within  budget. 
When  they  aren’t,  there  is  risk. 

•  Contracting  for  Risk  Management  helps 
ensure  the  people  involved  with  the  details 
of  the  technical  processes  of  design,  test,  and 
production  are  involved  with  managing  risk. 


The  principle  here  is  that  those  performing 
the  technical  details  are  normally  the  first  ones 
to  know  when  risks  exist. 

•  Quality  Standards,  such  as  IS09000, 
ANSI/ASQC  Q  9000,  MIL-HDBK  9000, 

and  others  describe  processes  for  develop¬ 
ing  and  producing  quality  products.  Com¬ 
paring  our  processes  with  these  standards  can 
highlight  areas  we  may  want  to  change  to 
avoid  risk. 

•  Use  of  Independent  Risk  Assessors  is  a 

method  to  help  ensure  all  risk  is  identified. 
The  knowledgeable,  experienced  people  are 
independent  from  the  management  and 
execution  of  the  processes  and  procedures 
being  reviewed.  Independent  assessment 
promotes  questions  and  observations  not 
otherwise  achievable. 

3.3.2.3  Analysis.  Risk  analysis  is  an  evalua¬ 
tion  of  the  identified  risk  events  to  determine 
possible  outcomes,  critical  process  variance 
from  known  best  practices,  the  probability/like¬ 
lihood  of  those  events  occurring,  and  the  con¬ 
sequences/impacts  of  the  outcomes.  Once  this 
information  has  been  determined,  the  risk  event 
may  be  rated  against  the  program’s  criteria  and 
an  overall  assessment  of  low,  moderate,  or  high 
assigned.  Figure  B-4  depicts  the  risk  analysis 
process  and  procedures. 

Critical  Process  Variance.  For  each  process  risk 
related  event  identified,  the  variance  of  the  pro¬ 
cess  from  known  standards  or  best  practices  must 
be  determined.  As  shown  in  Figure  B-4,  there 
are  five  levels  (a-e)  in  the  XYZ  risk  assessment 
process,  with  the  corresponding  criteria  of  Mini¬ 
mal,  Small,  Acceptable,  Large,  and  Significant. 
If  there  is  no  variance  then  there  is  no  risk. 

Probability/Likelihood.  For  each  risk  area  iden¬ 
tified,  the  probability/likelihood  the  risk  will 
happen  must  be  determined.  As  shown  in  Figure 
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B-4,  there  are  five  levels  (a-e)  in  the  XYZ  risk 
assessment  process,  with  the  corresponding  sub¬ 
jective  criteria  of  Remote,  Unlikely,  Likely, 
Highly  Likely,  and  Near  Certainty.  If  there  is  zero 
probability/likelihood  of  an  event,  there  is  no  risk 
per  our  definition. 

Consequence/impact.  For  each  risk  area  identi¬ 
fied,  the  following  question  must  be  answered: 
Given  the  event  occurs,  what  is  the  magnitude 
of  the  consequence/impact?  As  shown  in  the  fig¬ 
ure,  there  are  five  levels  of  consequence/impact 
(a-e).  “Consequence/impact”  is  a  multifaceted 
issue.  For  this  program,  there  are  four  areas  that 
we  will  evaluate  when  determining  conse¬ 
quence/impact:  technical  performance,  schedule, 
cost,  and  impact  on  other  teams.  At  least  one  of 
the  four  consequence/impact  areas  needs  to  ap¬ 
ply  for  there  to  be  risk;  if  there  is  no  adverse 


consequence/impact  in  any  of  the  areas,  there  is 
no  risk. 

•  Technical  Performance:  This  category  in¬ 
cludes  all  requirements  that  are  not  included 
in  the  other  three  metrics  of  the  Conse¬ 
quence/Impact  table.  The  wording  of  each 
level  is  oriented  toward  design  processes, 
production  processes,  life  cycle  support,  and 
to  retirement  of  the  system.  For  example,  the 
word  “margin”  could  apply  to  weight  mar¬ 
gin  during  design,  safety  margin  during  test¬ 
ing,  or  machine  performance  margin  during 
production. 

•  Schedule:  The  words  used  in  the  Schedule 
column,  as  in  all  columns  of  the  Consequence/ 
Impact  table,  are  meant  to  be  universally 
applied.  Avoid  excluding  a  consequence/ 
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Consequence 


RISK  ASSESSMENT 

HIGH — Unacceptable.  Major 
disruption  likely.  Different 
approach  required.  Priority 
management  attention 
required. 

MODERATE— Some 
disruption.  Different 
approach  may  be  required. 
Additional  management 
attention  may  be  needed. 

LOW — Minimum  impact. 
Minimum  oversight  needed 
to  ensure  risk  remains  low. 


Level 

Technical  and  / 

Performance  or 

and/ 

Schedule  or 

and/ 

Cost  or 

Impact  on 
Other  Teams 

a 

Minimal  or  no  impact 

Minimal  or  no  impact 

Minimal  or  no  impact 

None 

b 

Acceptable  with  some 
reduction  in  margin 
need  dates 

Additional  resources 
required;  able  to  meet 

<5% 

Some  impact 

c 

Acceptable  with  significant 
reduction  in  margin 

Minor  slip  in  key  milestones; 
not  able  to  meet  need  date 

5-7% 

Moderate  impact 

d 

Acceptable;  no  remaining 
margin 

Major  slip  in  key  milestone 
or  critical  path  impacted 

7-10% 

Major  impact 

e 

Unacceptable 

major  program  milestone 

Can't  achieve  key  team  or 

>10% 

Unacceptable 

Figure  B-4.  Risk  Assessment  Process 
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impact  level  from  consideration  just  because 
it  doesn’t  match  your  team’s  specific  defini¬ 
tions.  In  other  words,  phrases  such  as  need 
dates,  key  milestones,  critical  path,  and  key 
team  milestones  are  meant  to  apply  to  all  IPTs. 

•  Cost:  Since  costs  vary  from  component  to 
component  and  process  to  process,  the  per¬ 
centage  criteria  shown  in  the  figure  may  not 
strictly  apply  at  the  lower  levels  of  the  WBS. 
These  team  leaders  can  set  the  percentage 
criteria  that  best  reflects  their  situation.  How¬ 
ever,  when  costs  are  rolled  up  at  higher  levels 
(e.g.,  Program),  the  following  definitions  will 
be  used:  Level  1 — no  change,  Level  2 — 
<5%,  Level  3 — 5-7%,  Level  4 — 7-10%,  and 
Level  5 — >10%. 

•  Impact  on  Other  Teams:  Both  the  conse¬ 
quence/impact  of  a  risk  and  the  handling  ac¬ 
tions  associated  with  reducing  the  risk  may 
impact  another  team.  This  may  involve 
additional  coordination  or  management 
attention  (resources)  and  may  therefore 
increase  the  level  of  risk.  This  is  especially 
true  of  common  technical  processes. 

Risk  Rating.  Probability  and  consequence/ 
impact  should  not  always  be  considered  equally; 
for  example,  there  may  be  consequences/impacts 
so  severe  that  it  is  considered  high  risk  even 
though  the  probability  to  achieve  a  particular 
outcome  is  low.  After  deciding  a  level  of  pro¬ 
cess  variance/probability/likelihood  (a  through 
e)  and  a  level  of  consequence/impact  (a  through 
e),  enter  the  Assessment  Guide  portion  of  Fig¬ 
ure  B-4  to  obtain  a  risk  rating  (green  =  LOW, 
yellow  =  MOD,  and  red  =  HIGH).  For  example; 
consequence/impact/process  variance/probabil¬ 
ity/likelihood  level  2b  corresponds  to  LOW  risk, 
level  3d  corresponds  to  MOD  risk,  level  5c  cor¬ 
responds  to  HIGH  risk.  After  obtaining  the  risk 
rating,  make  a  subjective  comparison  of  the  risk 
event  with  the  applicable  rating  definition  in  Fig¬ 
ure  B-4  (e.g.,  High=  unacceptable,  major 


disruptions,  etc.).  There  should  be  a  close  match. 
If  there  isn’t,  consider  reevaluating  the  level  of 
probability /likelihood  or  consequence/impact. 
Those  risk  events  that  are  assessed  as  moderate 
or  high  should  be  submitted  to  the  XYZ  Risk 
Management  Coordinator  on  a  RIF. 

Figure  B-4  is  useful  to  convey  information  to 
decision  makers  and  will  be  used  primarily  for 
that  purpose.  The  PMO  will  use  the  Risk  Track¬ 
ing  Report  and  Watch  List.  (See  Annex  D.) 

3.4  RISK  HANDLING 

3.4.1  Process 

After  the  program’s  risks  have  been  identified 
and  assessed,  the  approach  to  handling  each  sig¬ 
nificant  risk  must  be  developed.  There  are 
essentially  four  techniques  or  options  for  han¬ 
dling  risks:  avoidance,  control,  transfer,  and 
assumption.  For  all  identified  risks,  the  various 
handling  techniques  should  be  evaluated  in  terms 
of  feasibility,  expected  effectiveness,  cost  and 
schedule  implications,  and  the  effect  on  the 
system’s  technical  performance,  and  the  most 
suitable  technique  selected.  Section  2524.3  of 
the  Defense  Acquisition  Deskbook  contains  in¬ 
formation  on  the  risk-handling  techniques  and 
various  actions  that  can  be  used  to  implement 
them.  The  results  of  the  evaluation  and  selec¬ 
tion  will  be  included  and  documented  in  the 
RMIS  using  the  RIF.  This  documentation  will 
include:  what  must  be  done,  the  level  of  effort 
and  materials  required,  the  estimated  cost  to 
implement  the  plan,  a  proposed  schedule  show¬ 
ing  the  proposed  start  date,  the  time  phasing  of 
significant  risk  reduction  activities,  the  comple¬ 
tion  date,  and  their  relationship  to  significant  Pro¬ 
gram  activities/milestones  (an  example  is  pro¬ 
vided  in  Annex  B),  recommended  metrics  for 
tracking  the  action,  a  list  of  all  assumptions,  and 
the  person  responsible  for  implementing  and 
tracking  the  selected  option. 
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3.4.2  Procedures 

The  IPT  that  assessed  the  risk  is  responsible  for 
evaluating  and  recommending  to  the  PM  the 
risk-handling  options  that  are  best  fitted  to  the 
program’s  circumstances.  Once  approved,  these 
are  included  in  the  program’s  acquisition  strategy 
or  management  plans,  as  appropriate. 

For  each  selected  handling  option,  the  responsible 
IPT  will  develop  specific  tasks  that,  when  imple¬ 
mented,  will  handle  the  risk.  The  task  descriptions 
should  explain  what  has  to  be  done,  the  level  of 
effort,  and  identify  necessary  resources.  It  should 
also  provide  a  proposed  schedule  to  accomplish 
the  actions  including  the  start  date,  the  time  phas¬ 
ing  of  significant  risk  reduction  activities,  the 
completion  date,  and  their  relationship  to  signifi¬ 
cant  Program  activities/milestones  (an  example  is 
provided  in  Annex  B),  and  a  cost  estimate.  The 
description  of  the  handling  options  should  list  all 
assumptions  used  in  the  development  of  the  han¬ 
dling  tasks.  Assumptions  should  be  included  in 
the  RIF.  Recommended  actions  that  require  re¬ 
sources  outside  the  scope  of  a  contract  or  official 
tasking  should  be  clearly  identified,  and  the  IPT s, 
the  risk  area,  or  other  handling  plans  that  may  be 
impacted  should  be  listed. 

Reducing  requirements  as  a  risk  avoidance  tech¬ 
nique  will  be  used  only  as  a  last  resort,  and  then 
only  with  the  participation  and  approval  of  the 
user’s  representative. 

DoD  4245 .7-M  Templates  and  NAVSO  P-6071 
Best  Practices  Manual ,  are  useful  in  develop¬ 
ing  risk-handling  actions  for  design,  test,  or 
manufacturing  process  risks. 

3.5  RISK  MONITORING 

3.5.1  Process 

Risk  monitoring  systematically  tracks  and  evalu¬ 
ates  the  performance  of  risk-handling  actions.  It 


is  part  of  the  PMO  function  and  responsibility 
and  will  not  become  a  separate  discipline.  Es¬ 
sentially,  it  compares  predicted  results  of  planned 
actions  with  the  results  actually  achieved  to  de¬ 
termine  status  and  the  need  for  any  change  in 
risk-handling  actions.  The  effectiveness  of  the 
risk-monitoring  process  depends  on  the  estab¬ 
lishment  of  a  management  indicator  system 
(metrics)  that  provides  accurate,  timely,  and  rel¬ 
evant  risk  information  in  a  clear,  easily  under¬ 
stood  manner.  (See  Annex  D.)  The  metrics  se¬ 
lected  to  monitor  program  status  must  adequately 
portray  the  true  state  of  the  risk  events  and  han¬ 
dling  actions.  Otherwise,  indicators  of  risks  that 
are  about  to  become  problems  will  go  undetec¬ 
ted. 

To  ensure  that  significant  risks  are  effectively 
monitored,  risk-handling  actions  (which  include 
specific  events,  schedules,  and  “success”  crite¬ 
ria)  will  be  reflected  in  integrated  program 
planning  and  scheduling.  Identifying  these  risk 
handling  actions  and  events  in  the  context  of 
Work  Breakdown  Structure  (WBS)  elements 
establishes  a  linkage  between  them  and  spe¬ 
cific  work  packages,  making  it  easier  to  deter¬ 
mine  the  impact  of  actions  on  cost,  schedule, 
and  performance.  The  detailed  information  on 
risk-handling  actions  and  events  will  be  in¬ 
cluded  in  the  RIF  for  each  identified  risk,  and 
thus  be  resident  in  the  RMIS. 

3.5.2  Procedures 

The  functioning  of  IPTs  is  crucial  to  effective 
risk  monitoring.  They  are  the  “front  line”  for 
obtaining  indications  that  risk-handling  efforts 
are  achieving  their  desired  effects.  Each  IPT  is 
responsible  for  monitoring  and  reporting  the  ef¬ 
fectiveness  of  the  handling  actions  for  the  risks 
assigned.  Overall  XYZ  program  risk  assessment 
reports  will  be  prepared  by  the  XYZ  Risk  Man¬ 
agement  Coordinator  working  with  the  cogni¬ 
zant  IPT. 
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Many  techniques  and  tools  are  available  for 
monitoring  the  effectiveness  of  risk-handling  ac¬ 
tions,  and  IPTs  must  ensure  that  they  select  those 
that  best  suit  their  needs.  No  single  technique  or 
tool  is  capable  of  providing  a  complete  answer — 
a  combination  must  be  used.  At  a  minimum,  each 
IPT  will  maintain  a  watch  list  of  identified  high 
priority  risks.  See  Section  2. 5. 2.4  of  the  Defense 
Acquisition  Deskbook  for  information  on  spe¬ 
cific  techniques. 

Risks  rated  as  Moderate  or  High  risk  will  be 
reported  to  the  XYZ  Risk  Management  Coor¬ 
dinator,  who  will  also  track  them,  using  infor¬ 
mation  provided  by  the  appropriate  IPT,  until 
the  risk  is  considered  Low  and  recommended 
for  “Close  Out.”  The  IPT  that  initially  reported 
the  risk  retains  ownership  and  cognizance  for 
reporting  status  and  keeping  the  database 
current.  Ownership  means  implementing  han¬ 
dling  plans  and  providing  periodic  status  of  the 
risk  and  of  the  handling  plans.  Risk  will  be 
made  an  agenda  item  at  each  management  or 
design  review,  providing  an  opportunity  for  all 
concerned  to  offer  suggestions  for  the  best 
approach  to  managing  risk.  Communicating 
risk  increases  the  program’s  credibility  and 
allows  early  actions  to  minimize  adverse 
consequences/impacts. 

The  risk  management  process  is  continuous. 
Information  obtained  from  the  monitoring  pro¬ 
cess  is  fed  back  for  reassessment  and  evalua¬ 
tions  of  handling  actions.  When  a  risk  area  is 
changed  to  Low,  it  is  put  into  a  “Historical  File” 
by  the  Risk  Management  Coordinator  and  it  is 
no  longer  tracked  by  the  XYZ  PMO.  The 
“owners”  of  all  Low  risk  areas  will  continue 
monitoring  Low  risks  to  ensure  they  stay  Low. 

The  status  of  the  risks  and  the  effectiveness  of 
the  risk-handling  actions  will  be  reported  to  the 
Risk  Management  Coordinator: 

•  Quarterly; 


•  When  the  IPT  determines  that  the  status  of 
the  risk  area  has  changed  significantly  (as  a 
minimum  when  the  risk  changes  from  high 
to  moderate  to  low,  or  vice  versa);  and 

•  When  requested  by  the  Program  Manager. 

4.0  RISK  MANAGEMENT 

INFORMATION  SYSTEM  (RMIS) 
AND  DOCUMENTATION 

The  XYZ  program  will  use  the  XXX  database 
management  system  as  its  RMIS.  The  system 
will  contain  all  of  the  information  necessary 
to  satisfy  the  program  documentation  and 
reporting  requirements. 

4.1  RISK  MANAGEMENT 
INFORMATION  SYSTEM 

The  RMIS  stores  and  allows  retrieval  of  risk- 
related  data.  It  provides  data  for  creating  reports 
and  serves  as  the  repository  for  all  current  and 
historical  information  related  to  risk.  This 
information  will  include  risk  assessment  docu¬ 
ments,  contract  deliverables,  if  appropriate,  and 
any  other  risk-related  reports.  The  PMO  will 
use  data  from  the  RMIS  to  create  reports  for 
senior  management  and  retrieve  data  for  day- 
to-day  management  of  the  program.  The  pro¬ 
gram  produces  a  set  of  standard  reports  for 
periodic  reporting  and  has  the  ability  to  create 
ad  hoc  reports  in  response  to  special  queries. 
See  Annex  D  for  a  detailed  discussion  of  the 
RMIS. 

Data  are  entered  into  the  RMIS  using  the  Risk 
Information  Form  (RIF).  The  RIF  gives  mem¬ 
bers  of  the  project  team,  both  Government  and 
contractors,  a  standard  format  for  reporting  risk- 
related  information.  The  RIF  should  be  used 
when  a  potential  risk  event  is  identified  and  will 
be  updated  as  information  becomes  available  as 
the  assessment,  handling,  and  monitoring  func¬ 
tions  are  executed. 
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4.2  RISK  DOCUMENTATION 

All  program  risk  management  information  will 
be  documented,  using  the  RIF  as  the  standard 
RMIS  data  entry  form.  The  following  paragraphs 
provide  guidance  on  documentation  require¬ 
ments  for  the  various  risk  management  functions. 

4.2.1  Risk  Assessment  Documentation 

Risk  assessments  form  the  basis  for  many  pro¬ 
gram  decisions.  From  time  to  time,  the  PM  will 
need  a  detailed  report  of  any  assessment  of  a 
risk  event.  It  is  critical  that  all  aspects  of  the 
risk  management  process  are  documented. 

4.2.2  Risk-Handling  Documentation 

Risk-handling  documentation  will  be  used  to  pro¬ 
vide  the  PM  with  the  information  he  needs  to 
choose  the  preferred  handling  option. 

4.2.3  Risk  Monitoring  Documentation 

The  PM  needs  a  summary  document  that  tracks 
the  status  of  high  and  moderate  risks.  The  Risk 
Management  Coordinator  will  produce  a  risk 
tracking  list,  for  example,  that  uses  information 
that  has  been  entered  from  the  RMIS.  This  docu¬ 
ment  will  be  produced  on  a  monthly  basis. 


4.3  REPORTS 

Reports  are  used  to  convey  information  to 
decision  makers  and  team  members  on  the 
status  of  the  program  and  the  effectiveness  of 
the  risk  management  program.  Every  effort  will 
be  made  to  generate  reports  using  the  data 
resident  in  the  RMIS. 

4.3.1  Standard  Reports 

The  RMIS  will  have  a  set  of  standard  reports.  If 
IPTs  or  functional  managers  need  additional 
reports,  they  should  work  with  the  Risk 
Management  Coordinator  to  create  them.  Access 
to  the  reporting  system  will  be  controlled; 
however,  any  member  of  the  Government  or 
contractor  team  may  obtain  a  password  to  gain 
access  to  the  information.  See  Annex  B  for  a 
description  of  the  XYZ  program  reports. 

4.3.2  Ad  Hoc  Reports 

In  addition  to  standard  reports,  the  PMO  will 
need  to  create  ad  hoc  reports  in  response  to 
special  queries.  The  Risk  Management  Coordi¬ 
nator  will  be  responsible  for  these  reports. 
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ANNEX  A 

TO  XYZ  RISK  MANAGEMENT  PLAN 
—  CRITICAL  PROGRAM  ATTRIBUTES  — 


Category 

Description 

Responsible  IPT 

Remarks 

Performance/Physical 

Speed 

Weight 

Endurance 

Crew  Size 

Survivability 

Maneuverability 

Size 

Receiver  Range 

Transmitter  Range 

Data  Link  Operations 

Recovery  Time 

Initial  Setup 

Identification  Time 

Accuracy  Location 

Probability  of  Accurate  ID 

Reliability 

Maintainability 

Availability 

Etc. 

Cost 

Operating  and  Support  Costs 

Etc. 

Processes 

Requirements  Stable 

Test  Plan  Approved 

Exit  Criteria 

Engine  Bench  Test 

Accuracy  Verified  by  Test  Data 
and  Analysis 

Toolproofing  Completed 

Logistics  Support  Reviewed  by 
User 

Table  B-1.  Critical  Program  Attributes 
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ANNEX  B 

TO  XYZ  RISK  MANAGEMENT  PLAN 
—  PROGRAM  RISK  REDUCTION  SCHEDULE  — 

(EXAMPLE  OF  RISK  HANDLING  PLAN  SCHEDULE) 


Accomplished 


Planned 


H 

I 

G 

H 


Determine  and  flowdown  requirements,  evaluate  potential  hardware  and  software  solutions.  Gather  data 
on  NDI  capabilities,  limitations,  evaluate  alternatives  and  pick  lower  risk  solutions. 


■  Simulations  to  evaluate  subsystem  interactions,  timing  issues.  Simulations  to  evaluate  target  sets, 
*  environment  effects. 


R 

I 

S 

K 

R 

A 

T 

I 

N 

G 


m  Preliminary  design  and  trade  studies  to  work  issues  such  as  temperature  and  shock  environments. 
^  Develop  baseline  design.  Reassess  risk. 


M 

E 

D 

I 

U 

M 


yf 


Get  hardware  and  software  in  place  for  pre-SDD  simulations.  Consolidate  team  structure  and 
supplier. 


■  Hardware-in-the-Loop  (HWIL)  and  performance  prediction  demo.  Supporting  analyses  and  design 
^  studies. 


'i' 

■  Initiate  detailed  trade  studies  and  identify  alternatives.  Validate  and  implement  trade  study 
.  decisions  with  customer  on  IPD  teams  for  lower  risk  options.  Reassess  risk. 


L 

O 

w 


■l  Extensive  simulations  &  HWIL  testing.  Developmental  test  program,  supporting 
J  analyses,  reviews  and  decisions. 

— - 1  Systems  integration  testing  (supported  by  continued  simulations)  to 

- 1  verify  design.  TAAF  program  with  selected  subsystems.  Reassess  risk. 

"jr _ 

Qualification  testing. 

■ 

_ 

|]  Operational  testing  &  simulations. 
;  (LRIP  items) 


MSB  MS  C  ^ - ^Production. 

\  SRR  SFR  PDR  CDR  \  FCA  PCA  FRP 

TV  v  V  VTVVV 


CR 

TD 

SDD 

P&D 

2000 

2001 

2002 

2003 

2004  2005  2006  2007 

Figure  B-5.  XYZ  Program  Risk  Handling  Plan  Schedule  (Example) 
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ANNEX  C 

TO  XYZ  RISK  MANAGEMENT  PLAN 
—  PROGRAM  METRIC  EXAMPLES  — 


Engineering 

Requirements 

Production 

Support 

Key  Design  Parameters 

•  Weight 

•  Size 

•  Endurance 

•  Range 

Design  Maturity 

•  Open  problems 
reports 

•  Number  of 
engineering  change 
proposals 

•  Number  of  drawings 
released 

•  Failure  activities 

Computer  Resource 

Utilization 

Etc. 

Requirements  Traceability 

Requirements  Stability 

Threat  Stability 

Design  Mission  Profile 

Manufacturing  Yields 

Incoming  Material  Yields 

Delinquent  Requisitions 

Unit  Production  Cost 

Process  Proofing 

Waste 

Personnel  Stability 

Special  Tools  and  Test 
Equipment 

Support  Infrastructure 
Footprint 

Manpower  Estimates 

Table  B-2.  Examples  of  Product-Related  Metrics 


Design 

Requirements 

Trade 

Studies 

Design 

Process 

Integrated  Test 
Plan 

Failure 

Reporting 

System 

Manufacturing 

Plan 

Development  of 
requirements 
traceability  plan 

Development  of 
specification  tree 
Specifications 
reviewed  for: 

•  Definition  of 
all  use 
environ¬ 
ments 

•  Definition  of 
all  functional 
requirements 
for  each 
mission 
performed 

Users  needs 
prioritized 

Alternative 
system  configu¬ 
rations  selected 

Test  methods 
selected 

Design  require¬ 
ments  stability 

Producibility 
analysis  con¬ 
ducted 

Design  analyzed 
for: 

•  Cost 

•  Parts 
reduction 

•  Manufac¬ 
turability 

•  Testability 

All  developmen¬ 
tal  tests  at 
system  and 
subsystem  level 
identified 

Identification  of 
who  will  do  test 
(Government, 
contractor, 
supplier) 

Contractor 
corporate-level 
management 
involved  in 
failure  reporting 
and  corrective 
action  process 

Responsibility  for 
analysis  and 
corrective  action 
assigned  to 
specific  indi¬ 
vidual  with  close¬ 
out  date 

Plan  documents 
methods  by 
which  design  to 
be  built 

Plan  contains 
sequence  and 
schedule  of 
events  at  con¬ 
tractor  and  sub¬ 
contractor  levels 
that  defines  use 
of  materials,  fab¬ 
rication  flow,  test 
equipment,  tools, 
facilities,  and 
personnel 

Reflects  manu¬ 
facturing  inclu¬ 
sion  in  design 
process.  In¬ 
cludes  identi¬ 
fication  and 
assessment  of 
design  facilities 

Table  B-3.  Examples  of  Process  Metrics 
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Cost 

Schedule 

Cost  variance 

Schedule  variance 

Cost  performance  index 

Schedule  performance  index 

Estimate  at  completion 

Design  schedule  performance 

Management  reserve 

Manufacturing  schedule  performance 

Test  schedule  performance 

Table  B-4.  Examples  of  Cost  and  Schedule  Metrics 
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ANNEX  D 

TO  XYZ  RISK  MANAGEMENT  PLAN 
—  MANAGEMENT  INFORMATION  SYSTEM  AND  DOCUMENTATION  — 


1.0  DESCRIPTION 

In  order  to  manage  risk,  we  need  a  database 
management  system  that  stores  and  allows  re¬ 
trieval  of  risk-related  data.  The  Risk  Manage¬ 
ment  Information  System  (RMIS)  provides  data 
for  creating  reports  and  serves  as  the  repository 
for  all  current  and  historical  information  related 
to  risk.  This  information  may  include  risk  as¬ 
sessment  documents,  contract  deliverables,  if 
appropriate,  and  any  other  risk-related  reports. 
The  Risk  Management  Coordinator  is  respon¬ 
sible  for  the  overall  maintenance  of  the  RMIS, 
and  he  or  his  designee  are  the  only  persons  who 
may  enter  data  into  the  database. 

The  RMIS  will  have  a  set  of  standard  reports. 
If  IPTs  or  functional  managers  need  additional 
reports,  they  should  work  with  the  Risk  Man¬ 
agement  Coordinator  to  create  them.  Access  to 
the  reporting  system  will  be  controlled;  however, 
any  member  of  the  Government  or  contractor 
team  may  obtain  a  password  to  gain  access  to 
the  information. 


In  addition  to  standard  reports,  the  PMO  will 
need  to  create  ad  hoc  reports  in  response  to  spe¬ 
cial  queries  etc.  The  Risk  Management  Coor¬ 
dinator  will  be  responsible  for  these  reports.  Fig¬ 
ure  B-6  shows  a  concept  for  a  management  and 
reporting  system. 

2.0  RISK  MANAGEMENT  REPORTS— 
XYZ  PROGRAM 

The  following  are  examples  of  basic  reports  that 
a  PMO  may  use  to  manage  its  risk  program. 
Each  office  should  coordinate  with  the  Risk 
Management  Coordinator  to  tailor  and  amplify 
them,  if  necessary,  to  meets  its  specific  needs. 

2.1  RISK  INFORMATION  FORM 

The  PMO  needs  a  document  that  serves  the  dual 
purpose  of  a  source  of  data  entry  information 
and  a  report  of  basic  information  for  the  IPTs, 
etc.  The  Risk  Information  Form  (RIF)  serves  this 
purpose.  It  gives  members  of  the  project  team, 


Figure  B-6.  Conceptual  Risk  Management  and  Reporting  System 
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both  Government  and  contractors,  a  format  for 
reporting  risk-related  information.  The  RIF 
should  be  used  when  a  potential  risk  event  is 
identified  and  updated  over  time  as  information 
becomes  available  and  the  status  changes.  As  a 
source  of  data  entry,  the  RIF  allows  the  data¬ 
base  administrator  to  control  entries.  The  format 
for  a  RIF  is  included  on  page  B-29. 

2.2  RISK  ASSESSMENT  REPORT 

Risk  assessments  form  the  basis  for  many  pro¬ 
gram  decisions,  and  the  PM  may  need  a  detailed 
report  of  assessments  of  a  risk  event  that  has 
been  done.  A  Risk  Assessment  Report  (RAR) 
is  prepared  by  the  team  that  assessed  a  risk  event 
and  amplifies  the  information  in  the  RIF.  It  docu¬ 
ments  the  identification,  analysis,  and  handling 
processes  and  results.  The  RAR  amplifies  the 
summary  contained  in  the  RIF,  is  the  basis  for 
developing  risk-handling  plans,  and  serves  as  a 
historical  recording  of  program  risk  assessment. 
Since  RARs  may  be  large  documents,  they  may 
be  stored  as  files.  RARs  should  include  infor¬ 
mation  that  links  it  to  the  appropriate  RIF. 

2.3  RISK-HANDLING 
DOCUMENTATION 

Risk-handling  documentation  may  be  used  to 
provide  the  PM  with  information  he  needs  to 
choose  the  preferred  handling  option  and  is  the 
basis  for  the  handling  plan  summary  contained 
in  the  RIF.  This  document  describes  the  exami¬ 
nation  process  for  risk-handling  options  and 
gives  the  basis  for  the  selection  of  the 


recommended  choice.  After  the  PM  chooses  an 
option,  the  rationale  for  that  choice  may  be  in¬ 
cluded.  There  should  be  a  time-phased  plan  for 
each  risk-handling  task.  Risk-handling  plans  are 
based  on  results  of  the  risk  assessment.  This  docu¬ 
ment  should  include  information  that  links  it  to 
the  appropriate  RIF. 

2.4  RISK  MONITORING 
DOCUMENTATION 

The  PM  needs  a  summary  document  that  tracks 
the  status  of  high  and  moderate  risks.  The  XYZ 
program  will  use  a  risk-tracking  list  that  con¬ 
tains  information  that  has  been  entered  from 
the  RIF.  An  example  of  the  tracking  report/list 
is  shown  on  page  B-30. 

3.0  DATABASE  MANAGEMENT 
SYSTEM  (DBMS) 

The  XYZ  Risk  Management  Information  Sys¬ 
tem  (RMIS)  provides  the  means  to  enter  and 
access  data,  control  access,  and  create  reports. 
Key  to  the  MIS  are  the  data  elements  that  reside 
in  the  database.  Listed  below  are  the  types  of 
risk  information  that  will  be  included  in  the 
database.  “Element”  is  the  title  of  the  database 
field;  “Description”  is  a  summary  of  the  field 
contents.  The  Risk  Management  Coordinator 
will  create  the  standard  reports  such  as,  the  RIF, 
Risk  Monitoring,  etc.  The  RMIS  also  has  the 
ability  to  create  “ad  hoc”  reports,  which  can  be 
designed  by  users  and  the  Risk  Management 
Coordinator. 
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Element 

Description 

Risk  Identification 
(ID)  Number 

Identifies  the  risk  and  is  a  critical  element  of  information,  assuming  that  a 
relational  database  will  be  used  by  the  PMO.  (Construct  the  ID  number  to 
identify  the  organization  responsible  for  oversight.) 

Risk  Event 

States  the  risk  event  and  identifies  it  with  a  descriptive  name.  The  statement 
and  risk  identification  number  will  always  be  associated  in  any  report. 

Priority 

Reflects  the  importance  of  this  risk  priority  assigned  by  the  PMO  compared  to 
all  other  risks,  e.g.,  a  one  (1)  indicates  the  highest  priority. 

Data  Submitted 

Gives  the  date  that  the  RIF  was  submitted. 

Major  System/ 
Component 

Identifies  the  major  system/component  based  on  the  WBS. 

Subsystem/ 
Functional  Area 

Identifies  the  pertinent  subsystem  or  component  based  on  the  WBS. 

Category 

Identifies  the  risk  as  technical/performance  cost  or  schedule  or  combination  of 
these. 

Statement  of  Risk 

Gives  a  concise  statement  (one  or  two  sentences)  or  the  risk. 

Description  of 

Risk 

Briefly  describes  the  risk.  Lists  the  key  processes  that  are  involved  in  the 
design,  development,  and  production  of  the  particular  system  or  subsystem.  If 
technical/performance,  includes  how  it  is  manifested  (e.g.,  design  and 
engineering,  manufacturing,  etc.) 

Key 

Parameters 

Identifies  the  key  parameter,  minimum  acceptable  value,  and  goal  value,  if 
appropriate.  Identifies  associated  subsystem  values  required  to  meet  the 
minimum  acceptable  value  and  describes  the  principal  events  planned  to 
demonstrate  that  the  minimum  value  has  been  met. 

Assessment 

States  if  an  assessment  has  been  done.  Cites  the  Risk  Assessment  Report,  if 
appropriate. 

Analyses 

Briefly  describes  the  analysis  done  to  assess  the  risk.  Includes  rationale  and 
basis  for  results. 

Probability  of 
Occurrence 

States  the  likelihood  of  the  event  occurring,  based  on  definitions  in  the 
program’s  Risk  Management  Plan. 

Consequence 

States  the  consequence  of  the  event,  if  it  occurs,  based  on  definitions  in  the 
program’s  Risk  Management  Plan. 

Time  Sensitivity 

Estimates  the  relative  urgency  for  implementing  the  risk-handling  option. 

Other  Affected 

Areas 

If  appropriate,  identifies  any  other  subsystem  or  process  that  this  risk  affects. 

Risk  Handling  Plans 

Briefly  describes  plans  to  mitigate  the  risk.  Refers  to  any  detailed  plans  that 
may  exist,  if  appropriate. 

Risk  Monitoring 
Activity 

Measures  using  metrics  for  tracking  progress  in  implementing  risk  handling 
plans  and  achieving  planned  results  for  risk  reduction. 

Status 

Briefly  reports  the  status  of  the  risk-handling  activities  and  outcomes  relevant 
to  any  risk  handling  milestones. 

Status  Due  Date 

Lists  date  of  the  status  report. 

Assignment 

Lists  individual  assigned  responsibility  for  handling  activities. 

Reported  By 

Records  name  and  phone  number  of  individual  who  reported  the  risk. 

Table  B-5.  DBMS  Elements 
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Risk  Information  Form 

Risk  Identification  Number  Date 

Risk  Event: 

Priority 

Major  System/Component/Functional  Area: 

Category: 

Statement  of  Risk: 

Description  of  Risk: 


Key  Parameters: 
Assessment: 

Analysis: 


Process  Variance 
Probability  of  Occurrence: 

Consequence: 

Time  Sensitivity: 

Other  Affected  Areas: 

Risk  Handling  Plans: 

Risk  Monitoring  Activity: 

Status 

Status  Date: 

Assignment:  Reported  By: 


Figure  B-7.  Risk  Information  Form 
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Risk  Tracking  Report 

(Example  Report) 

1.  Risk  Area  Status: 

Design  PF:  Hi  CF:  Hi 

Significant  Design  Risks: 

1.  Title:  System  Weight  PF:  Hi  CF:  Hi 

Risk  Event: 

Exceed  system  weight  by  1 0%;  decreasing  the  range  and  increasing  fuel 
consumption. 

Action: 

Examining  subsystems  to  determine  areas  where  weight  may  be  reduced. 
Reviewing  the  requirement.  Closely  watching  the  effect  on  reliability  and 
survivability. 

2.  Title:  Design 

Analysis  PF:  Hi  CF:  Hi 

Risk  Event: 

Failure  Modes,  Effects  and  Criticality  Analysis  (FMECA)  is  planned  too  late 
to  identify  and  correct  any  critical  single-point  failure  points  prior  to  design 
freeze. 

Action: 

Additional  resources  are  being  sought  to  expedite  performance  of  FMECA. 

II.  Risk  Area  Status:  Supportability  PF:  Hi  CF:  Mod/Hi 

1.  Title:  Operational  Support  PF:  Hi  CF:  Mod/Hi 

Risk  Event: 

Power  supply  subcontractor  is  in  financial  trouble  and  may  go  out  of 
business.  No  other  known  sources  exist. 

Action: 

Doing  trade  study  to  see  if  alternative  designs  have  a  broader  power  supply 
vendor  base.  Prime  contractor  is  negotiating  with  the  subcontractor  to  buy 
drawings  for  development  of  second  source. 

Figure  B-8.  Risk  Tracking  Report  Example 
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Potential  Risk 

Risk  Reduction 

Action 

Date 

Area 

Actions 

Code 

Due  Date 

Completed 

Explanation 

•  Accurately 

•  Use  multiple  finite 

SE03 

31  Aug  01 

predicting  shock 

element  codes  & 

environment 

simplified  numerical 

shipboard 

models  for  early 

equipment  will 

assessments. 

experience. 

•  Shock  test  simple 

SE03 

31  Aug  02 

isolated  deck,  and 
proposed  isolated 
structure  to  improve 
confidence  in 
predictions. 

•  Evaluating 

•  Concentrate  on 

SE031 

31  Aug  01 

acoustic  impact 

acoustic  modeling 

of  the  ship 

and  scale  testing  of 

systems  that  are 

technologies  not 

not  similar  to 

demonstrated 

previous  designs. 

successfully  in  large- 
scale  tests  or  full- 

scale  trials. 

•  Factor  acoustic 

SE032 

31  Aug  02 

signature  mitigation 
from  isolated  modular 
decks  into  system 
requirements. 

Continue  model  tests 
to  validate  predictions 
for  isolated  decks. 

Table  B-6.  Watch  List  Example 
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SAMPLE  RISK  MANAGEMENT  PLAN 
FOR  THE  ABC  PROGRAM  (ACAT  III,  IV) 


1.0  INTRODUCTION 
1.1  PURPOSE 

This  Risk  Management  Plan  (RMP)  presents 
the  process  for  implementing  the  comprehen¬ 
sive  and  proactive  management  of  risk  as  part 
of  the  overall  management  of  the  ABC  Program. 
Risk  management  is  a  program  management 
tool  to  handle  events  that  might  adversely 
impact  the  program,  thereby  increasing  the 
probability/likelihood  of  success.  This  RMP 
describes  a  management  tool  that  will: 

•  Serve  as  a  basis  for  identifying  alternatives 
to  achieve  cost,  schedule,  and  performance 
goals, 

•  Assist  in  making  decisions  on  budget  and 
funding  priorities, 

•  Provide  risk  information  for  Milestone 
decisions,  and 

•  Allow  monitoring  the  health  of  the  program 
as  it  proceeds. 

The  RMP  describes  methods  for  assessing  (iden¬ 
tifying  and  analyzing),  prioritizing,  and  moni¬ 
toring  risk  drivers;  developing  risk-handling  ap¬ 
proaches,  and  applying  adequate  resources  to 
handle  risk.  It  assigns  specific  responsibilities  for 
these  functions,  and  prescribes  the  document¬ 
ing,  monitoring,  and  reporting  processes  to  be 
followed. 

If  necessary,  this  RMP  will  be  updated  on  the 
following  occasions:  (1)  whenever  the  acquisi¬ 
tion  strategy  changes,  or  there  is  a  major  change 
in  program  emphasis;  (2)  in  preparation  for  ma¬ 
jor  decision  points;  (3)  in  preparation  for,  and 


immediately  following,  technical  audits  and  re¬ 
views;  (4)  concurrent  with  the  review  and  up¬ 
date  of  other  program  plans;  and  (5)  in 
preparation  for  a  POM  submission. 

2.0  PROGRAM  SUMMARY 

2.1  DESCRIPTION 

The  ABC  Program  is  an  ACAT  III  level  program 
that  was  initiated  in  response  to  the  NEW  COM 
Operational  Requirements  Document  (ORD) 
XXX,  dated  DD-MM-YYYY  (NOTE:  The 
ORD  is  being  replaced  by  the  Capability  De¬ 
velopment  Document  (CDD)).  The  program  will 
provide  an  ABC  communications  system  that 
will  be  the  common  system  (transmitter/receiver/ 
controller)  for  all  DoD  components  for  UHF  sat¬ 
ellite  communications.  All  DoD  systems  requir¬ 
ing  UHF  satellite  communications  procured  sub¬ 
sequent  to  Initial  Operational  Capability  (IOC) 
of  the  ABC  system  will  incorporate  it  to  meet 
their  needs.  The  Bx  Unmanned  Air  Vehicle  is 
the  lead  system  for  integration.  The  program  has 
completed  the  Systems  Integration  part  of  the 
System  Development  and  Demonstration  (SDD) 
Phase  and  is  preparing  for  an  Interim  Progress 
Review. 

The  system  will  be  acquired  using  off-the-shelf 
UHF  satellite  communications  systems.  During 
the  System  Integration  (SI)  part  of  the  System 
Development  and  Demonstration  (SDD)  Phase 
of  the  program,  two  contractors  delivered 
prototypes  of  their  systems.  One  is  a  ruggedized 
commercial  product  and  the  other  is  built  to 
military  specifications.  The  Government  tested 
both  systems  against  functional  and  perfor¬ 
mance  requirements  and  some  environmental 
extremes.  Although,  each  failed  portions  of  the 
tests,  both  were  evaluated  as  mature  enough  to 
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represent  an  acceptable  risk  for  proceeding  to 
the  System  Demonstration  part  of  the  SDD 
Phase  of  the  program. 

2.2  ACQUISITION  STRATEGY 

The  Government  will  invite  the  contractors  that 
participated  in  System  Integration  (SI)  Part  of  the 
System  Development  and  Demonstration  (SDD) 
Phase  of  the  program  to  submit  pro-posals  to  re¬ 
fine  their  approached  into  a  stable,  interoperable, 
producible,  supportable,  and  cost-effective  design; 
validate  the  manufacturing  or  production  process; 
and  demonstrate  system  capabilities  through  test¬ 
ing.  The  Government  will  select  one  of  the  two 
proposals  for  the  System  Demonstration  part  of 
the  SDD  Phase  of  the  program.  The  contractor, 
upon  demonstration  of  exit  criteria  (See  Annex 
A),  will  proceed  with  a  Low  Rate  Initial 
Production  (LRIP)  of  the  system. 

The  IOC  (20  systems)  for  the  ABC  system  is 
required  by  FY-02  to  support  the  fielding  of  the 
Bx  UAV.  Production  capacity  for  the  ABC  sys¬ 
tem  at  IOC  is  expected  to  be  20  units  per  month 
to  meet  the  demand  of  new  systems. 

2.3  PROGRAM  MANAGEMENT 
APPROACH 

The  ABC  Program  Manager  (PM)  reports  to  the 
Program  Director,  Satellite  Communications 
who  has  responsibility  for  all  satellite  communi¬ 
cations  systems.  The  ABC  Program  Office  (PO) 
is  composed  of  the  PM  and  one  assistant,  with 
matrix  support  from  the  systems  command  or¬ 
ganizations,  and  program  management  support 
from  an  external  contractor.  An  integrated  man¬ 
agement  approach  will  be  used  for  this  program. 
The  government  and  selected  contractor  will  have 
representation  on  Integrated  Product  Teams 
(IPTs)  that  will  focus  on  cost,  design,  test,  manu¬ 
facturing,  and  support  of  the  system.  The  PM 
chairs  the  government  IPT  that  develops  strate¬ 
gies  for  acquisition  and  contracts. 


3.0  RISK-RELATED  DEFINITIONS 

The  Defense  Acquisition  Deskbook,  Section 

2.5.2. 1  contains  the  definitions  for  risk,  risk  man¬ 
agement,  risk  events,  and  the  terms  associated 
with  risk  management  that  will  be  used  by  the 
ABC  PO.  Variation  and  clarification  of  defini¬ 
tions  that  appear  in  the  Defense  Acquisition 
Deskbook ,  as  they  are  used  in  the  ABC  program 
are  described  below. 

3.1  TECHNICAL  RISK 

This  is  the  risk  associated  with  the  evolution  of 
the  design,  production,  and  supportability  of  the 
ABC  system  affecting  the  level  of  performance 
necessary  to  meet  the  operational  requirements. 
The  contractor  and  subcontractors’  design,  test, 
and  production  processes  (process  risk)  influence 
the  technical  risk  and  the  nature  of  the  product 
as  depicted  in  the  various  levels  of  the  Work 
Breakdown  Structure  (product  risk).  Process 
risks  are  assessed  in  terms  of  process  variance 
fro  known  best  practices  and  potential  conse¬ 
quences/impacts  of  the  variance.  Product  risks 
are  assessed  in  terms  of  technical  performance 
measures  and  observed  variances  from  estab¬ 
lished  profiles. 

3.2  COST  RISK 

The  risk  associated  with  the  ability  of  the  pro¬ 
gram  to  achieve  its  life-cycle  cost  objectives. 
Two  risk  areas  bearing  on  cost  are  (1)  the  risk 
that  the  cost  estimates  and  objectives  are  accurate 
and  reasonable  and  (2)  the  risk  that  program 
execution  will  not  meet  the  cost  objectives  as  a 
result  of  a  failure  to  mitigate  technical  risks. 

3.3  RISK  RATINGS 

This  is  the  value  that  is  given  to  a  risk  event  (or 
the  program  overall)  based  on  the  analysis  of 
the  probability/likelihood  and  consequences/ 
impacts  of  the  event.  For  the  ABC  program,  risk 
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ratings  of  low,  moderate,  or  high  will  be  assigned 
based  on  the  criteria  in  Section  6.2. 

4.0  RISK  MANAGEMENT  STATUS 
AND  STRATEGY 

4.1  RISK  MANAGEMENT  STATUS 

As  a  result  of  the  Program  Definition  and  Risk 
Reduction  Phase,  the  overall  risk  of  the  ABC 
Program  for  Milestone  C  is  assessed  as  moder¬ 
ate,  but  acceptable.  Moderate  risk  functional  ar¬ 
eas  are  environmental  requirements;  form,  fit  and 
function;  integration;  manufacturing;  and  cost. 

4.2  RISK  MANAGEMENT  STRATEGY 

The  ABC  Program  risk  management  strategy  is 
to  handle  program  risks,  both  technical  and  non¬ 
technical,  before  they  become  problems,  caus¬ 
ing  serious  cost,  schedule,  or  performance  im¬ 
pacts.  This  strategy  is  an  integral  part  of  the 
Acquisition  Strategy  and  the  program  manage¬ 
ment  approach,  and  will  be  executed  primarily 
through  the  Government-Contractor  PIPT  orga¬ 
nization.  The  PIPTs  will  continuously  and 
proactively  assess  critical  areas  (especially  those 
listed  in  the  previous  paragraph)  to  identify  and 
analyze  specific  risks  and  will  develop  options 
to  mitigate  all  risks  designated  as  moderate  and 
high.  The  PIPTs  will  also  identify  the  resources 
required  to  implement  the  developed  risk-han¬ 
dling  options.  The  PM,  through  the  Program 


Level  Integrated  Product  Team  (PLIPT),  will  re¬ 
view  and  approve  the  PIPT  options.  Once  ap¬ 
proved,  the  options  will  be  incorporated  into  the 
program  integrated  master  plan  (IMP)  and  inte¬ 
grated  master  schedule  (IMS).  The  PIPTs  will 
monitor  the  effectiveness  of  the  selected  handling 
options,  and  adjust  the  risk  handling  approach 
as  necessary. 

IPTs  will  keep  risk  information  current  by  using 
the  risk  management  information  system  de¬ 
scribed  in  paragraph  6.5.  Risk  status  will  be 
reported  at  all  program  reviews.  As  new  infor¬ 
mation  becomes  available,  the  PO  and  contrac¬ 
tor  will  conduct  additional  reviews  to  ascertain 
if  new  risks  exit.  The  goal  is  to  be  continuously 
looking  to  the  future  for  areas  that  may  severely 
impact  the  program. 

5.0  RISK  MANAGEMENT 
ORGANIZATION 

5.1  PROGRAM  OFFICE 

The  ABC  Program  risk  management  organiza¬ 
tion  is  shown  in  Figure  B-9.  This  structure  is 
integrated  into  the  contractor  and  Government’s 
existing  organizations.  Program  Integrated  Prod¬ 
uct  Teams  (PIPTs)  will  be  formed  for  the  func¬ 
tional  areas  that  are  critical  to  the  success  of  the 
program.  All  functional  areas  not  covered  by  a 
PIPT  will  be  assessed  and  reviewed  by  the 
PLIPT  co-chaired  by  the  ABC  PM  and  contractor 


Figure  B-9.  ABC  Risk  Management  Organization 
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PM,  to  ensure  adequate  vigilance  against  emerg¬ 
ing  risk  areas.  Independent  risk  assessors  amy 
conduct  reviews,  when  directed  by  the  PM,  to 
ensure  the  interface  requirements  of  user  systems 
are  being  met  by  the  ABC  system  design. 

The  PM  the  is  overall  coordinator  of  Risk 
Management  Program  and  is  responsible  for: 

•  Maintaining  this  Risk  Management  Plan; 

•  Maintaining  the  Risk  Management  Database; 

•  Approving  risk- handling  options ; 

•  Incorporating  risk-handling  actions  into  the 
program  master  plan  and  schedule; 

•  Briefing  the  decision  makers  on  the  status  of 
ABC  Program  risk  efforts;  and 

•  Preparing  risk  briefings,  reports,  and  docu¬ 
ments  required  for  Program  Reviews  and  the 
acquisition  Milestone  decision  processes. 

PLIPT 

The  PLIPT  is  responsible  for  complying  with 
the  DoD  risk  management  policy  and  for  struc¬ 
turing  an  efficient  and  useful  ABC  risk  manage¬ 
ment  approach  and  supporting  the  Risk 
Management  Coordinator/PM  in  carrying  out  his 
responsibilities.  The  PM  and  contractor  PM  Co- 
Chair  the  PLIPT.  The  PLIPT  membership  may 
be  adjusted,  but  is  initially  established  as  the 
chairs  of  the  PIPTs,  a  representative  from  the 
joint  requirements  and  users’  office,  and  a 
representative  from  the  contractor.  It’s  main  ef¬ 
fort  is  integration  of  risk  assessments  performed 
by  various  program  IPTs. 

PIPTs 

The  program  IPTs,  or  PIPTs,  are  the  backbone 
of  the  program  risk  management  efforts.  They 


will  execute  the  following  responsibilities 
relative  to  their  functional  areas: 

•  Conduct  risk  assessments  and  develop  risk¬ 
handling  options,  to  include  handling  plans 
and  resources  required. 

•  Monitor  effectiveness  of  risk-handling  actions. 

•  Review  and  recommend  to  the  PM  changes 
in  the  overall  risk  management  approach 
based  on  lessons  learned. 

•  Update  the  risk  assessments  quarterly,  or  as 
directed. 

•  Ensure  information  in  the  Risk  Management 
Database  is  current. 

•  Prepare  risk  status  reports  in  their  areas  for 
all  Program  and  Design  Reviews. 

•  Ensure  Design/Build  Team  responsibilities  in¬ 
corporate  appropriate  risk  management  tasks. 

•  Coordinate  PIPT  risk  management  activities 
with  the  PLIPT. 

6.0  RISK  MANAGEMENT 

STRUCTURE  AND  PROCEDURES 

The  ABC  program  will  use  a  structured  risk 
management  approach  consisting  of  four  ele¬ 
ments:  planning,  assessment,  handling,  and 
monitoring.  These  elements  and  the  general  pro¬ 
cedures  to  be  used  for  each  of  them  are  described 
in  subsequent  paragraphs  of  this  section.  A  num¬ 
ber  of  guidance  documents  are  useful  in  address¬ 
ing  these  risk  management  elements,  and  should 
be  used  as  appropriate  by  each  PIPT.  Some  of 
these  documents  are  listed  below.  (This  list  is 
not  meant  to  be  complete.) 

•  Defense  Acquisition  Deskbook,  Section  2.5.2, 
Risk  Management, 
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•  DAU,  Risk  Management  Guide,  June  2003, 

•  AFMC  Pamphlet  63-101,  Risk  Management, 
9  July  1997,  and 

•  The  Navy ’s  Best  Practices  Manual,  NAVSO 
P-6071,  and  Top  Eleven  Ways  to  Manage 
Technical  Risk,  NAVSO  P-3686,  provide 
insight  into  best  practices  within  the  Naval 
Service. 

6.1  RISK  PLANNING 

Risk  planning  is  essential  for  the  execution  of 
a  successful  risk  management  program.  It  will 
be  done  continuously  by  all  PIPTs  as  an  inte¬ 
gral  part  of  normal  ABC  program  management. 
This  RMP  serves  as  the  basis  for  all  detailed 
risk  planning,  which  must  be  continuous.  The 
following  paragraphs  provide  direction  for  the 
PIPTs  on  the  conduct  of  risk  planning  for  this 
program. 

•  PIPT s  will  develop  an  organized  and  thorough 
approach  to  assess,  handle,  and  monitor  risks. 
It  will  assign  responsibilities  for  specific  risk 
management  actions  and  establish  internal  risk 
reporting  and  documentation  requirements. 
The  PLIPT  will  monitor  the  planning  activi¬ 
ties  of  the  PIPTs  to  ensure  that  they  are  con¬ 
sistent  with  this  RMP  and  that  appropriate 
revisions  to  this  plan  are  made  when  required 
to  reflect  significant  changes  resulting  from 
the  PIPT  planning  efforts. 

•  Each  PIPT  will  establish  metrics  that  will 
measure  the  effectiveness  of  their  planned 
risk-handling  options.  See  Annex  C  for  an 
example  of  metrics  that  may  be  used. 

•  Each  PIPT  will  identify  the  resources  re¬ 
quired  to  implement  the  risk  management 
actions.  These  resources  include  time, 
material,  personnel,  and  cost.  Training  is  a 
major  consideration.  All  PIPT  members 


should  receive  instruction  on  the  fundamen¬ 
tals  of  risk  management  and  special  train¬ 
ing  in  their  areas  of  responsibility,  if  neces¬ 
sary.  General  risk  management  training  will 
be  arranged  by  the  PO;  PIPT  leaders  will 
identify  any  specialized  training  needs. 

•  This  RMP  establishes  the  basic  documenta¬ 
tion  and  reporting  requirements  for  the 
program.  PIPTs  should  identify  any  addi¬ 
tional  requirements,  consistent  with  this 
RMP,  that  might  be  needed  to  effectively 
manage  risk  at  their  level. 

6.2  RISK  ASSESSMENT 

The  risk  assessment  process  includes  the  identi¬ 
fication  of  critical  risk  events/processes,  the 
analyses  of  these  events/processes  to  determine 
the  probability/likelihood  of  occurrence/process 
variance  and  consequences/impacts,  and  the  pri¬ 
ority  of  the  risks.  The  output  of  this  process  pro¬ 
vides  the  foundation  for  all  the  program  risk¬ 
handling  actions.  Therefore,  it  is  essential  that 
all  members  of  the  ABC  program  team  be  as  thor¬ 
ough  as  possible  when  identifying  and  analyzing 
risks.  In  addition  to  the  normal  areas  of  design , 
test,  manufacturing,  etc.,  PIPTs  must  identify 
and  analyze  the  risks  associated  with  such  areas 
as  manpower,  environmental  impact,  system 
safety  and  health  analysis,  and  security  consid¬ 
erations.  The  Defense  Acquisition  Deskbook, 
Section  2. 5. 2. 4,  provides  information  on  vari¬ 
ous  risk  assessment  techniques. 

Risk  assessments  should  be  done  by  the  PIPTs 
and  the  PLIPT  with  active  participation  of  both 
Government  and  contractor  personnel.  When 
necessary  or  appropriate,  the  PIPTs  and  the 
PLIPT  can  direct  a  contractor-only  assessment, 
or  conduct  a  Government  assessment.  PIPTs  and 
the  PLIPT  should  continually  assess  the  risks  in 
their  areas,  reviewing  critical  risk  areas,  risk  rat¬ 
ings  and  prioritization,  and  the  effectiveness  of 
risk-handling  actions  whenever  necessary  to 
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assess  progress.  The  assessment  process  will  be 
iterative,  with  each  assessment  building  on  the 
results  of  previous  assessments.  PIPTs  and  the 
PLIPT  will  use  the  current  assessment  baseline 
as  the  starting  point  for  their  initial  assessment 
during  this  phase.  This  baseline  is  a  combination 
of  the  risk  assessment  delivered  by  the  contrac¬ 
tors  as  part  of  the  Concept  Refinement  (CR) 
Phase,  the  PMO  process  risk  assessment  done 
before  Milestone  B,  and  the  post  award  Integrated 
Baseline  Review  (IBR).  Risk  assessments  will  be 
updated  and  the  results  presented  at  all  functional 
and  program  reviews,  with  a  final  update  for  this 
phase  prepared  not  later  than  six  months  prior  to 
the  next  scheduled  Milestone  decision. 

6.2.1  Risk  Identification 

Each  PIPT  will  review  all  aspects  of  their  func¬ 
tional  areas  to  determine  the  critical  events  that 
would  prevent  the  program  from  achieving  its 
objectives.  They  should  apply  the  knowledge, 
best  judgment  and  experience  of  the  PIPT 
members,  lessons  learned  from  similar  programs, 
and  the  opinion  of  subject-matter  experts  (SMEs) 
to  identify  these  risk  events.  PIPTs  should  fol¬ 
low  these  general  procedures  to  identify  risk 
events: 

•  Understand  the  requirements  and  the  pro¬ 
gram  performance  goals,  which  are  defined 
as  thresholds  and  objectives  (see  the  Interim 
Defense  Acquisition  Guidebook  (ID AG).  Un¬ 
derstand  the  operational  (functional  and  en¬ 
vironmental)  conditions  under  which  the  val¬ 
ues  must  be  achieved  as  described  in  the  De¬ 
sign  Reference  Mission  Profile.  The  ORD 
and  Acquisition  Program  Baseline  (APB) 
contain  Key  Performance  Parameters  (KPPs). 

•  Determine  technical/performance  risks 
related  to  engineering  and  manufacturing 
processes.  Identify  those  processes  that  are 
planned  or  needed  to  design,  develop, 
produce,  and  support  the  system.  Compare 


these  processes  with  industry  best  practices 
and  identify  any  variances  or  new,  untried 
processes.  These  variances  or  untried  prac¬ 
tices  are  sources  of  risk.  The  contractor 
should  review  the  processes  to  be  used  by 
its  subcontractors  to  ensure  they  are  consis¬ 
tent  with  best  industry  practices.  Table  4-2 
of  the  DAU  Risk  Management  Guide 
shows  some  of  the  specific  of  sources  of  pro¬ 
cess  risk,  and  should  be  used  by  the  PIPTs. 
NAVSO  P-6071,  Best  Practices  Manual , 
which  describes  risks  associated  with  design, 
test,  production,  facilities,  logistics,  manage¬ 
ment,  and  funding,  should  also  be  used  by 
the  PIPTs  to  identify  risks. 

•  Determine  technical/performance  risks 
associated  with  the  product  (the  ABC  com¬ 
munications  system)  in  the  following  criti¬ 
cal  risk  areas:  design  and  engineering,  tech¬ 
nology,  logistics,  concurrency,  and  manufac¬ 
turing.  The  design  and  manufacturing  PIPTs 
will  identify  the  contract  WBS  elements 
down  to  level  3,  and  evaluate  each  of  these 
elements  to  identify  risk  events.  They  will 
use  a  variety  of  methods  to  accomplish  this: 
review  of  similar  programs,  existing  program 
plans,  expert  opinion,  etc. 

•  Identify  schedule  risk.  Each  PIPT  will 
determine  the  schedule  risk  associated  with 
its  functional  area.  When  identifying  this 
schedule  risk,  they  will  consider  the  risk  that 
the  schedule  estimate  is  accurate,  and  the  risk 
that  the  established  schedule  can  be  met.  The 
PLIPT  will  monitor  the  development  of  the 
schedule  risk  in  each  PIPT,  and  consolidate 
these  risks  to  identify  overall  program 
schedule  risk. 

•  Identify  cost  risk.  Each  PIPT  will  determine 
the  cost  risk  associated  with  its  functional  area. 
They  will  identify  risks  associated  with  the 
accuracy  of  the  cost  estimates  developed  for 
their  areas,  and  the  risk  that  the  established 


B-36 


cost  objectives  will  be  met.  The  Cost  PIPT 
will  monitor  the  development  of  the  other 
PIPT  cost  risk  efforts,  and  consolidate  their 
risks  into  a  set  of  overall  program  cost  risks. 

•  All  identified  risks  will  be  documented  in  the 
RMIS,  with  a  statement  of  the  risk  and  a  de¬ 
scription  of  the  conditions  or  situations  caus¬ 
ing  concern  and  the  context  of  the  risk.  See 
Paragraph  6.4  for  guidance  on  documenting 
identified  risks. 

In  identifying  risks,  PIPTs  should  be  particu¬ 
larly  alert  for  the  following  indicators.  They  are 

common  sources  of  risk  for  all  programs,  and 

will  be  applicable  to  the  ABC  program. 

•  Requirements  that  are  not  clearly  stated  or 
stable, 

•  Failure  to  use  Best  Practices, 

•  Use  of  new  processes  materials,  or  applica¬ 
tions  of  existing  technologies, 

•  Use  of  processes  lacking  rigor  in  terms  of 
maturity,  documentation  of  established 
procedures,  and  validation, 

•  Insufficient  resources — the  people,  funds, 
schedule,  and  tools,  necessary  for  successful 
development,  test,  production  and  support  of 
the  ABC  program, 

•  Lack  of  a  formalized  failure,  reporting, 
analyze,  and  corrective  action  (FRACAS)  sys¬ 
tem, 

•  Use  of  suppliers  or  subcontractors  who  are 
inexperienced  in  the  processes  for  designing 
and  producing  required  products, 

•  Failure  of  prime  contractor  to  effectively 
monitor  processes  and  establish  quality  re¬ 
quirements  for  suppliers  and  subcontractors. 


6.2.2  Risk  Analysis 

Risk  Analysis  is  an  evaluation  of  the  identified 
risk  events  to  determine  the  probability  /likeli¬ 
hood  of  the  events  occurring  and  their  conse¬ 
quences/impacts,  to  assign  a  risk  rating  based 
on  the  program  criteria,  and  to  prioritize  the 
risks.  Each  PIPT  and  the  PLIPT  are  responsible 
for  analyzing  those  risk  events  they  identify. 
They  may  use  subject  matter  experts  for  assis¬ 
tance,  such  as  Field  Activities,  Service  Labo¬ 
ratories,  contractors,  or  outside  consultants.  The 
use  of  external  assets  will  be  coordinated 
through  the  PMO.  The  results  of  the  analysis 
of  all  identified  risks  must  be  documented  in 
the  RMIS. 

There  are  a  number  of  techniques  available  to 
support  risk  analysis,  to  include  studies,  test 
results,  modeling  and  simulation,  and  the 
opinions  of  qualified  experts  (to  include 
justification  of  their  judgment).  The  Defense 
Acquisition  Deskbook,  Section  2524.2  describes 
a  number  of  analysis  techniques  that  may  be 
useful.  Regardless  of  the  technique  used,  PIPTs 
and  the  PLIPT  will  identify  all  assumptions  made 
in  analyzing  risk  and,  where  appropriate,  conduct 
a  sensitivity  analysis  of  assumptions. 

Lor  each  risk  event,  the  following  risk  analysis 
guidelines  will  be  used: 

•  Probability/Likelihood 

Lor  each  risk  identified,  determine  the  probabil¬ 
ity/likelihood  that  the  event  will  occur.  Live 
levels  of  probability/likelihood  will  be  used  for 
the  ABC  program.  Table  B-7  shows  these  levels 
and  their  definitions.  PIPTs  and  the  PLIPT  will 
assign  one  of  these  values  to  each  identified 
risk  event  based  on  their  analysis  of  the  event. 
Lor  example,  if  it  is  known  that  there  will  be  a 
variance  between  the  soldering  process  to  be 
used  for  component  X  and  the  industry  standard, 
this  process  variance  risk  event  will  be  assigned 


B-37 


Level 

Likelihood  of  Occurrence 

a 

Remote 

b 

Unlikely 

c 

Likely 

d 

Highly  likely 

e 

Near  certainty 

Table  B-7.  Likelihood  Levels 


a  probability /likelihood  value  of  “e” — near  cer¬ 
tainty.  Similarly,  if  the  Manufacturing  PIPT 
determines  that  the  schedule  estimate  for  the  fab¬ 
rication  of  component  Y  is  overly  optimistic,  and 
will  probably  not  be  attained,  it  would  assign  a 
probability  /likelihood  level  of  “c”  or  “d”  de¬ 
pending  on  its  analysis  of  the  schedule  estimate. 

•  Consequence/Impact 

For  each  risk  identified,  the  following  question 
must  be  answered:  Given  the  event  occurs,  what 
is  the  magnitude  of  the  consequence/impact? 
For  the  ABC  program,  consequence/impact  will 
be  determined  in  each  of  four  areas:  technical 
performance,  schedule,  cost,  and  impact  on  other 
teams. 

Technical  Performance:  This  category  relates 
to  the  risks  associated  with  the  processes  to  be 
used  in  the  development,  testing,  and  manufac¬ 
turing  of  the  ABC  system,  and  the  nature  of  the 
ABC  communications  system.  It  includes  the 
form,  fit,  function,  manufacturability,  support- 
ability,  etc.  Essentially,  technical  risk  includes 
all  requirements  that  are  not  part  of  cost  and 
schedule.  The  wording  of  each  consequence/ 
impact  level  is  oriented  toward  design  and  pro¬ 
duction  processes,  life  cycle  support,  and  re¬ 
tirement  of  the  system.  For  example,  the  word 
“margin”  could  apply  to  weight  margin  during 
design,  safety  margin  during  testing,  or  machine 
performance  margin  during  production. 


Schedule:  The  description  in  the  Schedule  is 
self-explanatory.  The  need  dates,  key  mile¬ 
stones,  critical  path,  and  key  team  milestones  are 
meant  to  apply  to  all  program  areas  and  PIPTs. 

Cost:  Since  costs  vary  from  component  to  com¬ 
ponent  and  process  to  process,  the  percentage 
criteria  shown  in  the  figure  may  not  strictly 
apply  at  the  lower  levels  of  the  WBS.  PIPT  and 
PLIPT  leaders  may  set  the  percentage  criteria 
that  best  reflect  their  situation.  However,  when 
costs  are  rolled  up  at  higher  levels  (e.g.,  Pro¬ 
gram),  the  definitions  shown  will  be  used. 

Impact  on  Other  Teams:  Both  the  conse¬ 
quences/impacts  of  a  risk  and  the  handling  ac¬ 
tions  associated  with  addressing  the  risk  may  im¬ 
pact  another  team.  This  may  involve  additional 
coordination  or  management  attention  (re¬ 
sources),  and  may  therefore  increase  the  level 
of  risk.  This  is  especially  true  of  handling  actions 
that  involve  the  use  of  common  manufacturing 
processes  and/or  equipment. 

PIPTs  and  the  PLIPT  will  evaluate  each  risk 
event  in  terms  of  these  areas,  and  assign  a  level 
of  consequence/impact  (1-5).  Table  B-8  shows 
these  5  levels  of  consequence/impact,  and 
defines  the  levels  for  each  area.  This  table  will 
be  used  when  assigning  the  consequence/impact 
magnitude. 
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6.2.3  Risk  Rating 

Each  identified  risk  will  be  assigned  a  risk  rating 
based  on  the  joint  consideration  of  event  prob¬ 
ability/likelihood  and  consequence/impact. 
This  rating  is  a  reflection  of  the  severity  of  the 
risk  and  provides  a  starting  point  for  the  devel¬ 
opment  of  options  to  handle  the  risk.  It  is 
important  to  consider  both  the  probability/ 
likelihood  and  consequences/impacts  in  estab¬ 
lishing  the  rating,  for  there  may  be  risk  events 
that  have  a  low  probability/likelihood,  but 
whose  consequences/impacts  are  so  severe  that 
the  occurrence  of  the  event  would  be  disastrous 
to  the  program. 

Figure  B-10  describes  the  risk  rating  process 
that  will  be  used  in  this  program.  PIPTs  and 
the  PLIPT  will  analyze  each  risk  event  to  deter¬ 
mine  the  probability/likelihood  and  consequence/ 


impact  values  using  the  definitions  in  Tables  B- 
7  and  B-8;  they  will  determine  the  consequence/ 
impact  for  each  of  the  four  areas  (technical  per¬ 
formance,  schedule,  cost,  and  team  impact).  The 
values  will  be  used  to  determine  the  risk  rating 
using  the  Assessment  Guide  in  Figure  B-10.  The 
Assessment  Guide  defines  the  risk  rating  asso¬ 
ciated  with  each  combination  of  probability/like¬ 
lihood  and  consequence/impact  values,  and  will 
be  used  throughout  the  program.  For  example, 
consequence/impact/probability/likelihood  level 
lb  corresponds  to  a  risk  rating  of  (F)  FOW,  level 
4b  corresponds  to  MODERATE  risk,  and  level 
5c  corresponds  to  HIGH  risk. 

Those  risk  events  that  are  assessed  as  MOD¬ 
ERATE  or  HIGH  will  be  submitted  to  the  ABC 
PM  on  a  Risk  Identification  Form  (RIF).  See 
Appendix  B  for  the  RIF  format.  PIPTs  and 
the  PFIPT  must  actively  manage  these 


Level 

What  is  the  Likelihood  the 
Risk  Event  Will  Happen? 

a 

Remote 

b 

Unlikely 

c 

Likely 

d 

Highly  likely 

e 

Near  certainty 

ASSESSMENT  GUIDE 


* 


Process  Variance  refers  to  ^ 
I  deviation  from  best  practices. 

|  Likelihood/Probability  refers  to 
^isk  events.  j 
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Consequence 


RISK  ASSESSMENT 

HIGH — Unacceptable. 

Major  disruption  likely. 
Different  approach  required. 
Priority  management 
attention  required. 

MODERATE— Some 
disruption.  Different 
approach  may  be  required. 
Additional  management 
attention  may  be  needed. 

LOW — Minimum  impact. 
Minimum  oversight  needed 
to  ensure  risk  remains  low. 


Level 

Technical  and  / 

Performance  or 

and  / 

Schedule  or 

and  / 

Cost  or 

Impact  on 
Other  Teams 

a 

Minimal  or  no  impact 

Minimal  or  no  impact 

Minimal  or  no  impact 

None 

b 

Acceptable  with  some 
reduction  in  margin 
need  dates 

Additional  resources 
required;  able  to  meet 

<5% 

Some  impact 

c 

Acceptable  with  significant 
reduction  in  margin 

Minor  slip  in  key  milestones; 
not  able  to  meet  need  date 

5-7% 

Moderate  impact 

d 

Acceptable;  no  remaining 
margin 

Major  slip  in  key  milestone 
or  critical  path  impacted 

7-10% 

Major  impact 

e 

Unacceptable 

major  program  milestone 

Can’t  achieve  key  team  or 

>10% 

Unacceptable 

Figure  B-10.  Risk  Assessment  Process 
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Level 

Technical 

Performance 

Schedule 

Cost 

Impact  on 

Other  Teams 

a 

Minimal  or  no  impact 

Minimal  or  no  impact 

Minimal  or 
no  impact 

None 

b 

Acceptable  with  some 
reduction  in  margin 

Additional  resources 
required.  Able  to  meet 
need  dates 

<5% 

Some  impact 

c 

Acceptable  with 
significant  reduction 
in  margin 

Minor  slip  in  key  milestone. 
Not  able  to  meet  need  dates 

5-7% 

Moderate 

impact 

d 

Acceptable — no 
remaining  margin 

Major  slip  in  key  milestone 
or  critical  path  impacted 

7-1 0% 

Major  impact 

e 

Unacceptable 

Can’t  achieve  key  team  or 
major  program  milestone 

>10% 

Unacceptable 

Table  B-8.  Risk  Consequence 


MODERATE  and  HIGH  risks.  They  must  also 
continuously  assess  the  other  identified  risks  in 
their  areas  to  see  if  their  ratings  have  become 
MODERATE  or  HIGH. 

6.2.4  Risk  Prioritization 

PIPTs  and  the  PLIPT  will  prioritize  the  MOD¬ 
ERATE  and  HIGH  risks  in  their  areas.  This 
prioritization  will  provide  the  basis  for  the 
development  of  risk  handling  plans  and  the 
allocation  of  risk  management  resources. 
Prioritization  will  be  accomplished  using  expert 
opinion  within  the  PIPTs,  and  will  be  based  on 
the  following  criteria: 

•  Risk  Rating  -  Obviously  HIGH-MODER¬ 
ATE. 

•  Consequence/Impact  -  Within  each  rating, 
the  highest  value  of  consequence/impact,  e.g., 
“e.” 

•  Urgency  -  How  much  time  is  available  before 
risk-handling  actions  must  be  initiated. 


•  Probability/Likelihood  -  Within  each  rating , 
the  highest  value,  e.g.,  “e.” 

The  PLIPT  will  review  the  prioritized  list  of 
PIPT-developed  risks,  and  integrate  them  into 
a  single  list  of  prioritized  program  risks,  using 
the  same  criteria. 

6.3  RISK  HANDLING 

After  the  program’s  risks  have  been  identified, 
analyzed,  and  prioritized,  PIPTs  and  the  PLIPT 
must  develop  an  approach  for  handling  each 
MODERATE  and  HIGH  risk.  For  all  such  risks, 
the  various  handling  techniques  should  be 
evaluated  in  terms  of  feasibility,  expected 
effectiveness,  cost  and  schedule  implications, 
and  the  effect  on  the  system’s  technical  perfor¬ 
mance,  and  the  most  suitable  technique  selected. 
The  Defense  Acquisition  Deskbook,  Section 

2524.3  contains  information  on  the  risk-handling 
techniques  and  various  actions  that  can  be  used 
to  implement  them.  Reducing  requirements  as  a 
risk  avoidance  technique  will  be  used  only  as  a 
last  resort,  and  then  only  with  the  participation 
and  approval  of  the  user’s  representative  at  the 
PLIPT  level. 


B-40 


The  results  of  the  evaluation  and  selection  will 
be  included  and  documented  in  the  RMIS  using 
the  RIF.  This  documentation  will  include  the 
following  elements: 

•  What  must  be  done, 

•  List  of  all  assumptions, 

•  Level  of  effort  and  materials  required, 

•  Resources  needed  that  are  outside  the  scope 
of  the  contract  or  official  tasking, 

•  Estimated  cost  to  implement  the  plan, 

•  Proposed  schedule  showing  the  proposed 
start  date,  the  time  phasing  of  significant  risk 
reduction  activities,  the  completion  date,  and 
their  relationship  to  significant  Program 
activities/mile  stones , 

•  Recommended  metrics  for  tracking  risk¬ 
handling  activity, 

•  Other  PIPTs,  risk  areas,  or  other  handling 
plans  which  may  be  impacted,  and 

•  Person  responsible  for  implementing  and 
tracking  the  selected  option. 

Risk  handling  actions  will  be  integrated  into  pro¬ 
gram  planning  and  scheduling,  and  incorporated 
into  the  IMP  and  IMS.  PIPTs  and  the  PLIPT 
will  develop  these  risk-handling  actions  and 
events  in  the  context  of  Work  Breakdown  Struc¬ 
ture  (WBS)  elements,  establishing  a  linkage  be¬ 
tween  them  and  specific  work  packages  that 
makes  it  easier  to  determine  the  impact  of  ac¬ 
tions  on  cost,  schedule,  and  performance.  The 
detailed  information  on  risk-handling  actions  and 
events  will  be  included  in  the  RIF  for  each  iden¬ 
tified  risk,  and  thus  be  resident  in  the  RMIS. 


6.4  RISK  MONITORING 

Risk  monitoring  is  the  systematic  tracking  and 
evaluation  of  the  progress  and  effectiveness  of 
risk-handling  actions  by  the  comparison  of  pre¬ 
dicted  results  of  planned  actions  with  the  results 
actually  achieved  to  determine  status  and  the 
need  for  any  change  in  risk-handling  actions. 
The  PIPTs  and  the  PLIPT  will  monitor  all  iden¬ 
tified  risks  in  their  areas,  with  particular  atten¬ 
tion  to  those  rated  as  HIGH  or  MODERATE. 
There  are  a  number  of  techniques  and  tools 
available  for  monitoring  the  effectiveness  of  risk¬ 
handling  actions.  (See  the  Defense  Acquisition 
Deskbook,  Section  2524.4  for  information  on 
specific  techniques.)  PIPTs  and  the  PLIPT  must 
select  those  that  best  suit  their  needs.  No  single 
technique  or  tool  is  capable  of  providing  a  com¬ 
plete  answer — a  combination  must  be  used.  At 
a  minimum,  each  PIPT  and  the  PLIPT  will  use 
the  Risk  Tracking  Report  (RTR)  and  Watch  List 
for  day-to-day  management  and  monitoring  of 
risks.  See  Annex  B  for  examples  of  an  RTR  and 
Watch  List.  The  status  of  risk-handling  actions 
for  all  MODERATE  and  HIGH  risks  will  be  an 
agenda  item  at  each  program  or  functional  area 
review. 

For  each  identified  risk,  the  PIPTs  and  PLIPT 
will  establish  a  management  indicator  system 
(metrics)  that  provides  accurate,  timely,  and 
relevant  risk  monitoring  information  in  a  clear, 
easily  understood  manner.  PIPTs  and  the  PLIPT 
should  select  metrics  that  portray  the  true  state 
of  the  risk  events  and  handling  actions.  See  An¬ 
nex  C  for  an  example  of  metrics  that  may  be 
used. 

MODERATE  or  HIGH  risks  will  also  be  moni¬ 
tored  by  the  ABC  PM  through  the  PLIPT,  using 
information  provided  by  the  appropriate  PIPT, 
until  the  risk  is  considered  LOW  and  recom¬ 
mended  for  “Close  Out.”  PIPTs  and  the  PLIPT 
will  continue  to  monitor  LOW  risk  events  in 
their  areas  to  ensure  that  appropriate  risk- 
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handling  action  can  be  initiated  if  there  are 
indications  that  the  rating  may  change. 

The  status  of  the  risks  and  the  effectiveness  of 
the  risk-handling  actions  will  be  agenda  items 
for  all  functional  area  and  program  reviews,  and 
will  be  reported  to  the  PM  on  the  following  oc¬ 
casions: 

•  Quarterly, 

•  When  the  IPT  determines  that  the  status  of 
the  risk  area  has  changed  significantly  (as  a 
minimum  when  the  risk  changes  from  high 
to  moderate  to  low,  or  vice  versa), 

•  When  requested  by  the  Program  Manager. 

6.5  RISK  MANAGEMENT 

INFORMATION  SYSTEM  (RMIS), 
DOCUMENTATION,  AND 
REPORTS 

The  ABC  Program  uses  a  modified  version  of 
Risk  Matrix  as  its  RMIS.  The  Risk  Matrix 


database  will  contain  all  of  the  information 
necessary  to  satisfy  the  program  documenta¬ 
tion  and  reporting  requirements.  This  informa¬ 
tion  will  include  risk  assessment  documents, 
risk-handling  plans,  contract  deliverables,  if  ap¬ 
propriate,  and  any  other  risk-related  reports. 
The  program  office  will  use  data  from  the  RMIS 
to  create  reports  for  senior  management  and  for 
day-to-day  management  of  the  program.  The 
program  produces  a  set  of  standard  reports  for 
periodic  reporting  and  has  the  ability  to  create 
ad  hoc  reports  in  response  to  special  queries. 

Each  PIPT  and  the  PLIPT  are  responsible  for 
entering  and  maintaining  accurate  risk  manage¬ 
ment  data  in  the  RMIS.  A  standard  format  Risk 
Information  Form  (RIF)  Data  will  be  used  for 
data  entry.  A  RIF  will  be  completed  and  sub¬ 
mitted  when  a  potential  risk  event  is  identified, 
and  will  be  updated  as  information  becomes 
available  as  the  assessment,  handling,  and 
monitoring  functions  are  executed.  See  Annex 
B  for  a  sample  of  the  RIF.  Annex  B  also  con¬ 
tains  examples  of  reports  to  be  used  in  the  ABC 
Program. 
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ANNEX  A 

TO  ABC  RISK  MANAGEMENT  PLAN 
—  CRITICAL  PROGRAM  ATTRIBUTES  — 


Category 

Description 

Responsible  IPT 

Remarks 

Performance/Physical 

Transmitter  Power 

Weight 

MTBF 

Receiver  Gain 

EMP  Survivability 

Heat  Dissipation 

Size 

Receiver  Range 

Transmitter  Range 

Data  Link  Operations 

Interface  Commonality 

Initial  Setup 

Identification  Time 

Accuracy  Location 

Bandwidth 

Reliability 

Maintainability 

Availability 

Etc. 

Cost 

Operating  and  Support  Costs 

Etc. 

Processes 

Requirements  Stable 

Test  Plan  Approved 

Exit  Criteria 

Bench  Test 

Accuracy  Verified  by  Test  Data 
and  Analysis 

Toolproofing  Completed 

Logistics  Support  Reviewed  by 
User 

Table  B-9.  Critical  Program  Attributes 
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ANNEX  B 

TO  ABC  RISK  MANAGEMENT  PLAN 
— MANAGEMENT  INFORMATION  SYSTEM  AND  DOCUMENTATION — 


1.0  DESCRIPTION 

In  order  to  manage  risk,  we  need  a  database 
management  system  that  stores  and  allows 
retrieval  of  risk-related  data.  The  Risk  Manage¬ 
ment  Information  System  provides  data  for 
creating  reports  and  serves  as  the  repository  for 
all  current  and  historical  information  related  to 
risk.  The  PM  is  responsible  for  the  overall 
maintenance  of  the  RMIS,  and  he/she  or  his/ 
her  designee  are  the  only  persons  who  may  enter 
data  into  the  database. 

The  RMIS  has  a  set  of  standard  reports.  If  PIPTs 
or  functional  managers  need  additional  reports, 
they  should  work  with  the  PM  to  create  them. 
Access  to  the  reporting  system  will  be  con¬ 
trolled,  however  any  member  of  the  Government 
or  contractor  team  may  obtain  a  password  to  gain 
access  to  the  information. 

In  addition  to  standard  reports,  the  PO  will  need 
to  create  ad  hoc  reports  in  response  to  special 
queries,  etc.  The  PM  will  be  responsible  for  these 
reports. 

2.0  RISK  MANAGEMENT  FORMS 
AND  REPORTS 

The  following  are  examples  of  basic  reports  and 
forms  that  are  used  in  the  ABC  Program. 

2.1  RISK  INFORMATION  FORM 

The  PO  needs  a  document  that  serves  the  dual 
purpose  of  a  source  of  data  entry  information 
and  a  report  of  basic  information  for  the  PIPTs, 
etc.  The  Risk  Information  Form  (RIF)  serves 
this  purpose.  It  gives  members  of  the  project 


team,  both  Government  and  contractors,  a  format 
for  reporting  risk-related  information.  The  RIF 
will  be  used  when  a  potential  risk  event  is  iden¬ 
tified  and  updated  over  time  as  information  be¬ 
comes  available  and  the  status  changes.  As  a 
source  of  data  entry,  the  RIF  allows  the  data¬ 
base  administrator  to  control  entries.  The  format 
and  information  required  in  a  RIF  is  detailed  in 
Table  B-10,  Data  Base  Management  System 
(DBMS)  Elements. 

2.2  RISK  MONITORING 
DOCUMENTATION 

The  PM  needs  a  summary  document  that  tracks 
the  status  of  HIGH  and  MODERATE  risks.  The 
ABC  program  will  use  a  Risk-Tracking  Report 
(RTR)  that  contains  information  that  has  been 
entered  from  the  RIF.  An  example  of  the  RTR 
is  shown  in  Figure  B-ll.  The  PM  and  PIPTs 
must  also  be  aware  of  upcoming  deadlines  and 
events  to  ensure  they  are  not  caught  unprepared 
for  a  result.  A  Watch  List  will  be  used  to  track 
upcoming  events  and  activities.  A  sample  Watch 
List  is  contained  in  Table  B-ll. 

2.3  PIPT  RISK  SUMMARY  REPORT 

In  addition  to  the  RTRs  for  individual  HIGH 
and  MODERATE  risks,  PIPTs  will  prepare  a 
periodic  summary  of  the  ratings  for  all  the  risks 
in  their  areas .  Figure  B  - 1 2  provides  an  example 
of  this  report.  The  format  for  this  summary  is 
based  on  the  Risk  Assessment  Guide  shown  in 
Figure  B-10.  The  entries  in  each  cell  of  the 
matrix  represent  the  number  of  identified  risks 
with  the  corresponding  probability/likelihood 
and  consequence/impact  values. 
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Element 

Description 

Risk  Identification 
(ID)  Number 

Identifies  the  risk  and  is  a  critical  element  of  information,  assuming  that  a 
relational  database  will  be  used  by  the  PO.  (Construct  the  ID  number  to 
identify  the  organization  responsible  for  oversight.) 

Risk  Event 

States  the  risk  event  and  identifies  it  with  a  descriptive  name.  The  statement  and 
risk  identification  number  will  always  be  associated  in  any  report. 

Priority 

Reflects  the  importance  of  this  risk  priority  assigned  by  the  PO  compared  to  all 
other  risks,  e.g.,  a  one  (1)  indicates  the  highest  priority. 

Data  Submitted 

Gives  the  date  that  the  RIF  was  submitted. 

Major  System/Com¬ 
ponent  or  Process 

Identifies  the  major  system/component  based  on  the  WBS,  or  the  process  in 
which  the  risk  event  occurs. 

Subsystem/ 
Functional  Area 

Identifies  the  pertinent  subsystem  or  component  based  on  the  WBS. 

Category 

Identifies  the  risk  as  technical/performance  cost  or  schedule  or  combination  of 
these. 

Statement  of  Risk 

Gives  a  concise  statement  (one  or  two  sentences)  or  the  risk. 

Description  of 

Risk 

Briefly  describes  the  risk;  lists  the  key  processes  that  are  involved  in  the  design, 
development,  and  production  of  the  particular  system  or  subsystem.  If  technical/ 
performance,  include  how  it  is  manifested  (e.g.,  design  and  engineering, 
manufacturing,  etc.). 

Key  parameters 

Identifies  the  key  parameter,  minimum  acceptable  value,  and  goal  value,  if 
appropriate.  Identifies  associated  subsystem  values  required  to  meet  the 
minimum  acceptable  value  and  describes  the  principal  events  planned  to 
demonstrate  that  the  minimum  value  has  been  met. 

Assessment 

States  if  an  assessment  has  been  done.  Cites  the  Risk  Assessment  Report  (see 
next  paragraph),  if  appropriate. 

Analysis 

Briefly  describes  the  analysis  done  to  assess  the  risk;  includes  rationale  and 
basis  for  results. 

Process  Variance 

States  the  variance  of  critical  technical  processes  from  known  standards  or  best 
practices,  based  on  definitions  in  the  program’s  risk  management  plan. 

Probability  of 
Occurrence 

States  the  likelihood  of  the  event  occurring,  based  on  definitions  in  the 
program’s  Risk  Management  Plan. 

Consequence 

States  the  consequence  of  the  event,  if  it  occurs,  based  on  definitions  in  the 
program’s  Risk  Management  Plan. 

Risk  Rating 

Identifies  the  rating  assigned  to  the  risk  based  on  the  criteria  established  by  the 
program. 

Time  Sensitivity 

Estimates  the  relative  urgency  for  implement  the  risk-handling  option.  If 
appropriate,  identifies  any  other  subsystem  or  process  that  this  risk  affects. 

Other  Affected 
Areas 

If  appropriate,  identifies  any  other  subsystem  or  process  that  this  risk  affects. 

Risk  Handling 

Plans 

Briefly  describes  plans  to  mitigate  the  risk.  Refers  to  any  detailed  plans  that  may 
exist,  if  appropriate. 

Risk  Monitoring 
Activity 

Measurement  and  metrics  for  tracking  progress  in  implementing  risk-handling 
plans  and  achieving  planned  results  for  risk  reduction. 

Status 

Briefly  reports  the  status  of  the  risk-handling  activities  and  outcomes  relevant 
to  any  risk  handling  milestones. 

Status  Due  Date 

Lists  date  of  the  status  report. 

Assignment 

Lists  individual  assigned  responsibility  for  handling  activities. 

Reported  By 

Records  name  and  phone  number  of  individual  who  reported  the  risk. 

Table  B-10.  DBMS  Elements 
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Risk  Tracking  Report 

(Example  Report) 

1.  Risk  Area  Status: 

Design  PF:  Hi  CF:  Hi 

Significant  Design  Risks: 

1.  Title:  System  Weight  PF:  Hi  CF:  Hi 

Risk  Event: 

Exceed  system  weight  by  1 0%;  decreasing  the  range  and  increasing  fuel 
consumption. 

Action: 

Examining  subsystems  to  determine  areas  where  weight  may  be  reduced. 
Reviewing  the  requirement.  Closely  watching  the  effect  on  reliability  and 
survivability. 

2.  Title:  Design 

Analysis  PF:  Hi  CF:  Hi 

Risk  Event: 

Failure  Modes,  Effects  and  Criticality  Analysis  (FMECA)  is  planned  too  late 
to  identify  and  correct  any  critical  single-point  failure  points  prior  to  design 
freeze. 

Action: 

Additional  resources  are  being  sought  to  expedite  performance  of  FMECA. 

II.  Risk  Area  Status:  Supportability  PF:  Hi  CF:  Mod/Hi 

1.  Title:  Operational  Support  PF:  Hi  CF:  Mod/Hi 

Risk  Event: 

Power  supply  subcontractor  is  in  financial  trouble  and  may  go  out  of 
business.  No  other  known  sources  exist. 

Action: 

Doing  trade  study  to  see  if  alternative  designs  have  a  broader  power  supply 
vendor  base.  Prime  contractor  is  negotiating  with  the  subcontractor  to  buy 
drawings  for  development  of  second  source. 

Figure  B-11.  Example  Risk  Tracking  Report 
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Potential  Risk 

Risk  Handling 

Action 

Date 

Area 

Actions 

Code 

Due  Date 

Completed 

Explanation 

•  Accurately 

•  Use  multiple  finite 

SE03 

31  Aug  01 

predicting  shock 

element  codes  & 

environment 

simplified  numerical 

shipboard 

models  for  early 

equipment  will 

assessments. 

experience. 

•  Shock  test  simple 

SE03 

31  Aug  02 

isolated  deck,  and 
proposed  isolated 
structure  to  improve 
confidence  in 
predictions. 

•  Evaluating 

•  Concentrate  on 

SE031 

31  Apr  01 

acoustic  impact 

acoustic  modeling 

of  the  ship 

and  scale  testing  of 

systems  that  are 

technologies  not 

not  similar  to 

demonstrated 

previous  designs. 

successfully  in  large- 
scale  tests  or  full- 

scale  trials. 

•  Factor  acoustic 

SE032 

31  Aug  02 

signature  mitigation 
from  isolated  modular 
decks  into  system 
requirements. 

Continue  model  tests 
to  validate  predictions 
for  isolated  decks. 

Table  B-11.  Sample  Watch  List 
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Figure  B-12.  Example  PIPT  Risk  Summary  Report 
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Design 

Requirements 

Trade 

Studies 

Design 

Process 

Integrated  Test 
Plan 

Failure 

Reporting 

System 

Manufacturing 

Plan 

Development  of 
requirements 
traceability  plan 

Development  of 
specification  tree 

Specifications 
reviewed  for: 

•  Definition  of 
all  use 
environ¬ 
ments 

•  Definition  of 
all  functional 
requirements 
for  each 
mission 
performed 

Users  needs 
prioritized 

Alternative 
system  configu¬ 
rations  selected 

Test  methods 
selected 

Design  require¬ 
ments  stability 

Producibility 
analysis  con¬ 
ducted 

Design  analyzed 
for: 

•  Cost 

•  Parts 
reduction 

•  Manufac¬ 
turability 

•  Testability 

All  developmental 
tests  at  system 
and  subsystem 
level  identified 

Identification  of 
who  will  to  test 
(Government, 
contractor, 
supplier)  of 
requirements 
traceability  plan 

Development  of 
specification  tree 

Specifications 
reviewed  for: 

•  Definition  of 
all  use 

environments 

•  Definition  of 
all  functional 
requirements 
for  each 
mission 
performed 

Contractor 
corporate-level 
management 
involved  in 
failure  reporting 
and  corrective 
action  process 

Responsibility 
for  analysis  and 
corrective  action 
assigned  to 
specific  indivi¬ 
dual  with  close¬ 
out  date 

Plan  documents 
methods  by 
which  design  to 
be  built 

Plan  contains 
sequence  and 
schedule  of 
events  at 
contractor  and 
sub-contractor 
levels  that 
defines  use  of 
materials, 
fabrication  flow, 
test  equipment, 
tools,  facilities, 
and  personnel 

Reflects  manu¬ 
facturing 
inclusion  in 
design  process. 
Includes 

identification  and 
assessment  of 
design  facilities 

Table  B-13.  Examples  of  Process  Metrics 


Cost 

Schedule 

Cost  variance 

Schedule  variance 

Cost  performance  index 

Schedule  performance  index 

Estimate  at  completion 

Design  schedule  performance 

Management  reserve 

Manufacturing  schedule  performance 

Test  schedule  performance 

Table  B-14.  Example  of  Cost  and  Schedule  Metrics 
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APPENDIX  C 

GLOSSARY 


ACAT  -  Acquisition  Category 
AHP  -  Analytical  Hierarchy  Process 
AMSAA  -  Army  Materiel  System  Analysis  Activity 
APB  -  Acquisition  Program  Baseline 
API/PM  -  Acquisition  Program  Integration/Program  Management 
ASP  -  Acquisition  System  Protection 

BCS  -  Baseline  Comparison  System 
BIT  -  Built-in  Test 
BMP  -  Best  Manufacturing  Program 

CAIG  -  Cost  Analysis  Improvement  Group 
CAIV  -  Cost  As  an  Independent  Variable 
CARD  -  Cost  Analysis  Requirements  Description 
CCA  -  Component  Cost  Analysis 
CCDR  -  Contractor  Cost  Data  Reporting 
CDD  -  Capability  Development  Document 
CDF  -  Cumulative  Distribution  Function 
CDR  -  Critical  Design  Review 
CER  -  Cost  Estimating  Relationship 
CPM  -  Critical  Path  Method 
CR  -  Concept  Refinement 
CTD  -  Concept  and  Technology  Development 
CWBS  -  Contract  Work  Breakdown  Structure 

DAD  -  Defense  Acquisition  Deskbook 
DAU  -  Defense  Acquisition  University 
DBMS  -  Database  Management  System 
DCMA  -  Defense  Contract  Management  Agency 
DFARS  -  Defense  Federal  Acquisition  Regulation  Supplement 
DoD  -  Department  of  Defense 


C-l 


DoDD 

DoDI 

DPG 

DR 

DSMC 

DT&E 

DTSE&E 

EAC 

EMP 

ESC 

ESM 

ESS 

EV 

FMECA 

FRACAS 

GAO 

GFE 

HWIL 

IBR 

ICD 

IFF 

IIPT 

IMP 

IMS 

IOC 

IPD 

IPPD 

IPR 

IPT 

KPP 


DoD  Directive 
DoD  Instruction 
Defense  Planning  Guidance 
Decision  Review 

Defense  Systems  Management  College 

Development,  Test  and  Evaluation 

Director,  Test,  Systems  Engineering,  and  Evaluation 

Estimate  At  Completion 
Electromagnetic  Pulse 
Electronic  Systems  Center 
Electronic  Warfare  Support  Measures 
Environmental  Stress  Screening 
Earned  Value 

Failure  Mode,  Effects  and  Criticality  Analysis 
Failure,  Reporting,  Analyze,  and  Corrective  Action 

Government  Accounting  Office 
Government  Furnished  Equipment 

Hardware-in-the-Loop 

Integrated  Baseline  Review 

Initial  Capabilities  Document 

Identification  Friend  or  Foe 

Integrating  Integrated  Product  Teams 

Integrated  Master  Plan 

Integrated  Master  Schedule 

Initial  Operational  Capability 

Integrated  Product  Development 

Integrated  Product  and  Process  Development 

Interim  Progress  Review 

Integrated  Product  Teams 

Key  Perfoimance  Parameters 
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LCC 

LFT&E 

LRIP 

M&E 

M&S 

MAIS 

MDA 

MDAPs 

MIS 

MNS 

MOA 

MOU 

MS 

MTBF 

NDI 

NSSN 

O&M 

OIPT 

OLRDB 

ORD 

OSD 

OT&E 

P&D 

PDF 

PIPT 

PLIPT 

PM 

PMI 

PMO 

PMWS 

POE 


Life-Cycle  Cost 
Live-Fire  Test  and  Evaluation 
Low  Rate  Initial  Production 

Mechanical  and  Electrical 
Modeling  and  Simulation 
Major  Automated  Information  System 
Milestone  Decision  Authority 
Major  Defense  Acquisition  Programs 
Management  Information  System 
Mission  Need  Statement 
Memoranda  of  Agreement 
Memoranda  of  Understanding 
Milestone 

Mean  Time  Between  Failure 

Non-Developmental  Item 
New  Nuclear  Submarine 

Operations  and  Maintenance 
Overarching  Integrated  Product  Team 
On-Line  Risk  Data  Base 
Operational  Requirement  Document 
Office  of  the  Secretary  of  Defense 
Operational  Test  and  Evaluation 

Production  and  Deployment 
Probability  Density  Function 
Program  Integrated  Product  Team 
Program  Level  Integrated  Product  Team 
Program  Manager 
Project  Management  Institute 
Program  Management  Office 
Program  Manager’s  Work  Station 
Program  Office  Estimate 


C-3 


POM 

PRAG 

PRR 

PSR 

R&D 

R&M 

RD&A 

RAR 

RFP 

RIF 

RMIS 

RMP 

RTR 

SDD 

SEI 

SI 

SME 

SOW 

SPMN 

SRE 

SRR 

STA 

STAR 

T&E 

TAAF 

TD 

TEMP 

TPM 

TRIMS 

UAV 

UHF 

use 

USD(AT&L) 


Program  Objective  Memorandum 
Performance  Risk  Assessment  Group 
Production  Readiness  Review 
Program  Status  Report 

Research  and  Development 
Repairability  and  Maintainability 
Research,  Development  and  Acquisition 
Risk  Assessment  Report 
Request  for  Proposal 
Risk  Information  Form 
Risk  Management  Information  System 
Risk  Management  Plan 
Risk  Tracking  Report 

System  Development  and  Demonstration 
Software  Engineering  Institute 
System  Integration 
Subject-Matter  Expert 
Statement  of  Work 

Software  Program  Managers  Network 
Software  Risk  Evaluation 
System  Requirements  Review 
Special  Threat  Assessment 
Special  Threat  Assessment  Report 

Test  and  Evaluation 

Test,  Analyze,  and  Fix 

Technology  Development 

Test  and  Evaluation  Master  Plan 

Technical  Performance  Measurement 

Technical  Risk  Identification  and  Mitigation  Software 

Unmanned  Aerial  Vehicle 
Ultra-High  Frequency 
United  States  Code 

Under  Secretary  of  Defense,  Acquisition,  Technology,  and  Logistics 
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WBS  -  Work  Breakdown  Structure 
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APPENDIX  D 


QUANTIFYING 
EXPERT  JUDGMENT 


L  GENERAL 

Most  quantitative  risk  analysis  techniques  share 
a  common  need,  and  that  is  the  estimation  of  a 
probability  of  occurrence  associated  with  a  risk 
event.  Often  the  estimation  of  probability  data 
requires  expert  judgement,  and  inherent  in 
judgement  is  a  degree  of  uncertainty. 

The  challenge  for  the  analyst  is  to  obtain  esti¬ 
mates  in  the  areas  of  cost,  schedule,  and/or  tech¬ 
nical/performance.  These  estimates  often  begin 
as  qualitative  information  which  must  then  be 
converted  to  quantitative  probability  data  so  that 
the  results  can  be  represented  as  a  probability 
density  function  (PDF),  which  is  a  key  input  to 
a  number  of  different  types  of  models  (e.g., 
Monte  Carlo  simulations). 

There  are  a  number  of  methods  which  can  be 
used  to  convert  qualitative  estimates  into  quan¬ 
titative  probability  distributions.  The  remain¬ 
der  of  this  appendix  will  focus  on  a  few  of  the 
most  popular,  practical,  and  accurate  techniques 
for  doing  so.  The  techniques  discussed  were 
selected  because  they  are  relatively  simple  and 
easy  to  master.  This  factor  is  of  paramount  im¬ 
portance,  because  in  most  cases  the  analyst  and 
those  being  interviewed  will  have  neither  the 
time  nor  the  knowledge  of  more  advanced  tech¬ 
niques  to  accurately  implement  them.  Finally, 
the  use  of  these  techniques  does  not  preclude 
generating  uncertain  and/or  erroneous  PDFs  — 
the  quality  of  the  resulting  probability  distri¬ 
butions  will  be  no  better  than  the  interviewing 
technique  used  by  the  analyst,  the  level  of 
knowledge  of  the  experts  interviewed,  and  the 


ability  of  the  analyst  to  convert  the  information 
gleaned  from  participants  into  probability  distri¬ 
butions. 

The  following  techniques  will  be  discussed  in 
this  appendix: 

1.  Diagrammatic 

2.  Direct 

3.  Betting 

4.  Modified  Churchman/ Ackoff  technique 

5.  Delphi  Approach 

II.  DESCRIPTION  OF  TECHNIQUES 
1.  Diagrammatic 

Many  analysts  prefer  the  diagrammatic  method 
as  a  way  of  capturing  and  representing  an 
expert’s  judgement.  This  method  is  a  simple 
way  of  describing  an  expert’s  uncertainty  by 
presenting  him  with  a  range  of  PDF  diagrams 
and  having  the  expert  select  the  shape  of  the 
PDF  which  is  considered  to  reflect  most  accu¬ 
rately  the  schedule,  cost,  or  technical  param¬ 
eter  in  question.  Using  this  method,  the  analyst 
can  ascertain  whether  the  PDF  is  symmetric  or 
skewed,  the  degree  of  variability,  etc.  For  ex¬ 
ample,  if  the  expert  feels  that  there  is  a  great 
amount  of  risk  associated  with  completing  an 
activity  within  a  certain  period  of  time,  a  PDF 
skewed  to  the  right  may  be  selected.  Likewise, 
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activities  with  little  risk  may  be  skewed  to  the 
left.  If  the  expert  feels  that  each  value  over  a 
given  range  is  equally  likely  to  occur,  a  uniform 
distribution  may  be  most  appropriate.  The  ana¬ 
lyst  and  the  expert,  working  together,  can  select 
the  PDF  which  most  accurately  reflect  the 
schedule,  cost,  or  technical  item  under  question. 

The  diagrammatic  method  of  obtaining  PDFs 
is  applicable  when  the  expert  has  a  sound 
understanding  of  probability  concepts  and  can 
merge  that  understanding  with  his  understand¬ 
ing  of  the  parameters  under  question.  In  this 
way  the  expert  can  accurately  identify  the 
appropriate  PDFs. 

2.  Direct 

The  direct  method  is  a  relatively  simple  tech¬ 
nique  which  can  be  used  to  obtain  subjective 
probability  distributions  by  asking  the  expert 
to  assign  probabilities  to  a  given  range  of  values. 

The  direct  method  of  obtaining  PDFs  is  appli¬ 
cable,  1)  when  questions  can  be  phrased  to  the 
respondents  in  such  a  way  that  there  is  no  con¬ 
fusion  likely  to  exist  in  the  respondents  mind, 
and  2)  when  the  results  will  not  violate  the 
axioms  of  probability.  This  method  is  applicable 
when  time/resource  constraints  do  not  allow  for 
more  complex,  resource  intensive  methods. 

The  application  of  the  direct  method  is  quite 
simple.  The  analyst  would  define  a  relevant 
range  and  discrete  intervals  for  the  parameter 
for  which  the  PDF  is  to  be  constructed.  For 
example,  the  analyst  might  define  the  relevant 
time  duration  for  a  program  activity  (test  of  a 
piece  of  equipment)  to  be  between  0  and  27 
days.  The  analyst  would  then  break  this  relevant 
range  down  into  intervals,  say  intervals  of  three 
days,  the  resulting  formulation  would  look  as 
follows: 


0-3  days  16 -19  days 

4-7  days  20  -  23  days 

8-11  days  24  -  27  days 

12  -  15  days 

Given  these  intervals  over  the  relevant  range, 
the  analyst  would  then  query  the  expert  to  assign 
relative  probabilities  to  each  range.  From  this, 
the  form  of  the  PDF  could  be  identified.  It  is 
imperative  that  the  axioms  of  probability  not 
be  violated. 

Besides  the  application  already  described,  the 
analyst  could  request  the  expert  to  provide  a 
lowest  possible  value,  a  most  likely  value,  and 
a  highest  possible  value.  The  analyst  then  makes 
an  assumption  about  the  form  of  the  density 
function.  That  is,  is  the  PDF  uniform,  normal, 
beta,  triangular,  etc.? 

3.  Betting 

One  method  of  phrasing  questions  to  experts  in 
order  to  obtain  probabilities  for  ranges  of  values 
(cost/schedule)  states  the  problem  in  terms  of 
betting.  A  form  of  this  method,  which  was 
described  by  Winkler  (1967),  helps  the  expert 
(assessor)  assess  probabilities  of  events  which 
are  in  accordance  with  his  judgement.  The 
assumption  with  this  method  is  that  the  judge¬ 
ment  of  the  expert  may  be  fully  represented  by 
a  probability  distribution,  f(x)  of  a  random 
variable  x.  This  method  offers  the  expert  a  series 
of  bets. 

Under  ideal  circumstances,  the  bets  are  actual, 
not  hypothetical.  That  is,  in  each  case  the  win¬ 
ner  of  the  bet  is  determined  and  the  amount  of 
money  involved  actually  changes  hands.  How¬ 
ever,  under  our  circumstances,  this  is  not  fea¬ 
sible  (or  legal!).  In  each  case,  the  expert  must 
choose  between  two  bets  (the  expert  is  not 
allowed  to  refrain  from  betting).  The  expert 
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must  choose  between  a  bet  with  a  fixed  prob¬ 
ability  q  of  winning  and  1  -q  of  losing,  and  a 
bet  dependent  on  whether  or  not  some  event  E 
(a  particular  program  activity  duration  range, 
or  cost  range)  occurs.  The  bet  can  be  depicted 
as  follows: 

Bet  la  -  win  $A  if  the  event  E  occurs 

-  lose  $B  if  event  E  does  not  occur 

Bet  lb  -  win  $A  with  probability  of  q 

-  lose  $B  with  probability  of  l-q. 

The  expected  values  of  bets  la  and  lb  to  the 
expert  are  respectively  Ap  +  Bp  -  B  and  Aq  + 
Bq  -  B ,  where  P  is  the  probability  of  event  E 
occurring.  The  following  inferences  may  be 
drawn  from  the  experts  decision:  if  bet  la  is 
chosen,  Ap  +  Bp  -  B  >  Aq  +  Bq  -  B ,  so  p  >  q\ 
likewise  if  lb  is  selected  p  <  q. 

By  repeating  the  procedure,  varying  the  value 
of  q ,  the  probability  of  event  e  can  be  ascer¬ 
tained.  It  is  the  point  at  which  the  expert  is  in¬ 
different  between  bets  la  and  lb,  where  p  =  q. 
The  degree  of  precision  is  dependent  on  the 
number  of  bets  and  the  incremental  changes  of 
the  value  of  q. 

A  way  of  avoiding  the  problem  of  a  large  num¬ 
ber  of  bets  to  obtain  p  would  be  to  assess  the 
probabilities  through  the  use  of  direct  interro¬ 
gation,  and  then  to  use  the  betting  situation  as 
a  check  on  the  assumed  probabilities. 

To  complete  a  PDF,  the  analyst  repeats  this 
procedure  over  a  relevant  range  of  interval 
values.  The  analyst  then  plots  the  points  at  the 
center  of  the  range  for  each  event  and  smoothes 
in  a  curve,  so  that  the  area  under  it  equals  one, 
as  in  Figure  D-l.  The  analyst  must  ensure  that 
all  of  the  relevant  axioms  of  probability  are 
maintained. 


COST 


Figure  D-1. 

Fitting  a  Curve  to  Expert  Judgment 


Many  people,  when  questioned  one  way,  are 
likely  to  make  probability  statements  that  are 
inconsistent  with  what  they  will  say  when  ques¬ 
tioned  in  another  equivalent  way,  especially 
when  they  are  asked  for  direct  assignment  of 
probabilities.  As  the  number  of  events  increases, 
so  does  the  difficulty  of  assigning  direct  prob¬ 
abilities.  Therefore,  when  this  is  a  problem,  the 
betting  method  is  most  appropriate. 

To  apply  the  betting  technique,  we  will  select 
one  interval  for  the  relevant  range  to  demon¬ 
strate  how  this  method  can  be  used  to  obtain 
probability  estimates  and,  hence,  PDFs.  The  bet 
is  established  as  follows: 

Bet  la  -  win  $10,000  if  cost  is  between 
$15,000  and  $20,000 

-  lost  $5,000  if  cost  in  not  between 
$25,000  and  $20,000 

Bet  lb  -  win  $10,000  with  probability  of  q 

-  lose  $5,000  with  probability  of  l-q 

The  value  of  q  is  established  initially,  and  the 
expert  is  asked  which  of  the  two  bets  he  would 
take. 
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The  value  of  q  is  then  varied  systematically,  ei¬ 
ther  increased  or  decreased.  The  point  at  which 
the  expert  is  indifferent  between  the  two  bets 
(with  the  associated  q  value)  provides  the  prob¬ 
ability  of  the  cost  being  between  $15,000  and 
$20,000.  This  process  is  repeated  for  each  inter¬ 
val,  and  the  results  used  create  the  PDF  associ¬ 
ated  with  the  cost  of  that  particular  program 
event. 

4.  Modified  Churchman/Ackoff  Technique 

Another  method,  which  can  be  used  to  ascer¬ 
tain  PDFs  for  cost,  schedule,  or  performance 
parameters,  is  the  “Modified  Churchman/ 
Ackoff  method.”  This  technique  builds  upon 
procedures  which  were  presented  by  Church¬ 
man  and  Ackoff  in  1954.  This  technique  was 
developed  as  a  means  to  order  events  in  terms 
of  likelihood.  The  modification  to  the  technique 
was  performed  so  that  once  the  order  of  event 
likelihoods  had  been  accomplished,  relative 
probabilities  could  be  assigned  to  the  events  and 
finally  probability  density  functions  developed. 
So  as  to  be  relevant  for  our  purposes,  events 
are  defined  as  range  values  for  cost,  schedule, 
or  performance  (activity  durations)  relating  to  the 
outcome  of  a  specific  activity  in  a  program. 

The  modified  Churchman/Ackoff  technique  is 
most  appropriate  when  there  is  one  expert,  and 
that  expert  has  a  thorough  understanding  of  the 
relative  ranking  of  cost/schedule  ranges  and  a 
limited  understanding  or  probability  concepts. 
The  remainder  of  this  section  was  extracted  and 
modified  from  the  Compendium  on  Risk  Analy¬ 
sis  Techniques  (1972,  see  references).  Note  that 
while  the  mathematical  calculations  appear  to 
make  this  a  very  precise  technique,  it  is  still  an 
approximation  of  an  expert’s  judgement  and 
should  not  be  interpreted  to  be  more  exact  than 
other  similar  techniques. 

The  first  step  in  applying  the  modified  Church¬ 
man/Ackoff  technique  is  to  define  the  relevant 


range  of  values.  That  is,  the  end  points,  along  a 
range  of  values  with  zero  probability  of  occur¬ 
rence  must  be  specified.  These  values  need  only 
be  any  low  and  high  values  which  the  expert 
specifies  as  having  zero  probability  of  occur¬ 
rence.  Next,  ranges  of  individual  values  within 
the  relevant  range  must  be  determined.  These 
ranges  of  values  which  will  form  the  set  of  com¬ 
parative  values  for  this  technique  are  specified 
by  the  following  approach: 

(1)  Start  with  the  low  value  in  the  relevant 
range. 

(2)  Progress  upward  on  the  scale  of  values  until 
the  expert  is  able  to  state  a  simple  prefer¬ 
ence  regarding  the  relative  probabilities  of 
occurrence  of  the  two  characteristic  values. 
If  he  is  able  to  say  that  he  believes  one  value 
has  either  a  greater  chance  or  a  lesser  change 
of  occurring  than  the  other  of  the  two  values, 
then  it  is  inferred  that  the  expert  is  able  to 
discriminate  between  the  two  values. 

(3)  Using  the  higher  of  the  two  previously  speci¬ 
fied  scale  values  as  a  new  basis,  repeat  step 
(2)  to  determine  the  next  value  on  the  scale. 

(4)  Repeat  steps  (2)  and  (3)  until  the  high  end 
point  value  of  the  range  of  parameters 
values  is  approached. 


Employing  this  procedure  for  the  duration  re¬ 
quired  to  successfully  test  apiece  of  equipment, 
may  yield  the  results  show  in  Table  D-l. 


Table  D-1.  Characteristic  Values  for 
Equipment  Test  Durations 


The  descending  order  of  probability  or  occur¬ 
rence  can  be  determined  by  applying  the 
following  paired  comparison  method. 

Ask  the  expert  to  compare,  one  at  a  time,  the 
first  interval  value  (Oj)  of  the  set  to  each  of  the 
other  values  (02,  03,  etc.),  stating  a  preference 
for  that  value  in  each  group  of  two  values  that 
he  believes  has  the  greater  change  of  occurring 
(denoting  a  greater  probability  of  occurrence 
by  >,  and  equal  chance  by  =,  and  a  lesser  change 
by  <).  The  following  hypothetical  preference 
relationships  could  result  for  a  set  of  seven 
values  (Oj  <  02,  0|  <  03,  Oj  <  04,  Oj  <  05,  Oj  < 
06,  03  <  07). 

Next,  ask  the  expert  to  compare,  one  at  a  time, 
the  second  interval  values  (02)  of  the  set  to  each 
of  the  other  interval  values  succeeding  it  in  the 
set  (i.e.,  03,  04,  etc.).  The  following  preference 
relationships  might  result  (02  <  03,  02  <  04,  02 
<  05,  02  <  06,  02  <  07).  Continue  this  process 
until  all  values  (Oj)  have  been  compared. 


Now  total  the  number  of  times  (Oj)  value  was 
preferred  over  other  values.  The  results  for  this 
procedure  are  listed  in  Table  D-2. 


04 

= 

6  times 

03 

= 

5  times 

05 

= 

4  times 

02 

= 

3  times 

06 

= 

2  times 

0i 

= 

0  times 

07 

= 

0  times 

Table  D-2. 

Summary  of  Preference  Relationships 


List  the  values  in  descending  order  of  simple 
ordinal  probability  preference  and  change  the 
symbols  for  each  value  from  Oj  to  Xj  as  shown 
in  Table  D-3. 

Arbitrarily  assign  a  rating  of  100  points  to  the 
characteristic  value  with  the  highest  subjective 


Characteristic 

Value 

(Days) 

Preference 

Rank 

New 

Symbol 

0-3  04 

1 

XI 

4-7  03 

2 

X2 

8-11  05 

3 

X3 

04 

O 

in 

1 

CM 

4 

X4 

16-19  06 

5 

X5 

20  -  23  0j 

6 

X6 

24  -  27  07 

7 

X7 

Table  D-3. 
Transformation 


probability  (e.g.,  Xj).  Then,  as  in  the  first  step, 
question  the  expert  regarding  the  relative  chance 
of  occurrence  of  each  of  the  other  values  on 
the  ordinal  scale  in  Table  D-3  with  respect  to 
the  value  at  the  top  of  the  scale.  Assigning  Xl  a 
rating  of  100  points,  the  expert  is  first  interro¬ 
gated  as  to  his  feeling  of  the  relative  chance  of 
occurrence  of  the  second  highest  scale  value 
(e.g.,  X2),  with  respect  to  Xj.  Does  it  have  25 
percent  chance?  60  percent?  70  percent?  80 
percent?  As  much  chance  of  realization  as  X  j? 
The  relative  probability  rating,  based  on  100 
points  (i.e.,  100  percent  as  much  chance),  will 
then  be  posted  for  X2. 

Next,  question  the  expert  about  the  relative 
chance  of  occurrence  of  the  next  highest  scale 
(e.g.,  X3)  first  with  respect  to  the  most  preferred 
value  (XI),  and  then  with  respect  to  the  second 
most  preferred  scale  value  (X2).  The  resulting 
numerical  ratings  should  concur.  For  example, 
if  the  expert  decides  that  X2  has  8/10  as  much 
chance  of  occurring  as  does  Xj,  and  that  X3 
has  1/2  as  much  chance  as  X  1;  and  5/8  as  much 
chance  as  X2,  the  ratings  become  X3  =  100 
points,  X2  =  80  points,  and  X3  =  50  points. 

This  process  continues  for  each  successively 
lower  interval  value  on  the  ordinal  scale  as 
shown  in  Table  D-3.  Determine  the  relative 
number  of  points  to  be  accorded  each  value  with 
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respect  to  the  top  scale  and  with  respect  to  all 
other  values  on  down  the  scale  which  are  above 
the  characteristic  value  in  question. 

In  the  event  of  minor  disparities  between  rela¬ 
tive  probability  ratings  for  a  given  value,  the 
average  of  all  such  ratings  for  that  characteris¬ 
tic  value  might  be  computed.  For  example,  X4 
might  be  determined  to  be  3/10  as  probable  as 
Xj,  1/4  as  probable  as  X2,  and  1/2  as  probable 
as  X3.  The  three  absolute  ratings  for  X4  are  thus 
inferred  to  be  30, 20,  and  25  points,  respectively. 
The  average  of  these  ratings  is  25.  However, 
before  averaging  such  figures,  it  might  be  ben¬ 
eficial  to  have  the  expert  revaluate  his  relative 
ratings  for  X4  with  respect  to  X,,  X2,  and  X5. 

As  a  result  of  the  above  process,  the  relative 
probability  values  shown  in  Table  D-4  might 
be  attained. 


RXt 

= 

100 

Probability  points 

CM 

X 

IT 

= 

80 

Probability  points 

rx3 

= 

50 

Probability  points 

rx4 

= 

25 

Probability  points 

rx5 

= 

10 

Probability  points 

rx6 

= 

0 

Probability  points 

rx7 

0 

Probability  points 

Table  D-4. 

Relative  Probability  Ratings 


Similarly  P(Xi)  is  defined  as: 


R(Xj) 

R(Xj) 


[P(X,)J 


for  i  =  2,  3,  ...7. 

Assuming  that  the  independent  characteristic 
values  evaluated  represent  all  possible  values 
attainable  by  the  component  characteristic,  the 
respective  probabilities  must  sum  to  1.0  (i.e., 
P(Xj)  +  P(X2)  +  P(X3)  +  P(X4)  +  P(X5)  +  P(X6) 
+  P(X7)  =1.0).  Substituting  the  expressions  for 
P/Xj),  i  =  2,  . .  .7,  it  follows  that: 

(P(X-)I+S)[P(Xl,]+lir![P(Xl,]+^IP(X,)] 

+  ^1)[P(X1)]+  ^  [P(X,)]+  [P(Xj>]  =  1. 

R(Xj)  R(Xj)  R(Xj) 

Solving  this  equation  for  P(X  j ),  the  remaining 
P/Xj),  i  =  2,  ...7  can  be  determined  using  the 
relationship: 

p<x‘>  =  Hi 

As  an  illustration,  consider  the  relative  prob¬ 
ability  ratings  in  Table  D-4.  Using  the  values, 
the  preceding  equation  is  given  by: 


Finally,  the  scale  of  relative  probability  values 
can  be  converted  directly  into  a  scale  of  actual 
probability  density  values  by  letting  P(X  | )  equal 
the  actual  subjective  probability  or  occurrence 
of  the  highest  value.  Then,  P(X2)  is  then  defined 
as: 


80 

P(Xj)  + 

50 

■  P(Xj)  + 

too 

too 

25 

too 

P(Xj)  + 

10 

100 

PCX,)-  1 

Solving  this  equation,  P/X^  =  0.377. 


R(X2) 

R(Xi) 


[P(Xj)] 


This  value  can  be  used  to  determine  the  remaining 
probabilities  as  follows: 
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P(X2)  + 


rx7 

RXi 


P(Xj)  =  0.80  (0.377)  =  0.301 


P(X3)  + 


rx3 

rx7 


P(Xj)  =  0.50  (0.377)  =  0.189 


P(X4)  + 


rx4 

RX, 


P(Xj)  =  0.25  (0.377)  =  0.095 


known  as  the  Delphi  to  avoid  the  pressures. 

The  Delphi  technique  has  become  well  known 
in  management  circles,  but  is  subject  to  mis¬ 
conception.  Too  often  the  term  is  used  to  iden¬ 
tify  a  committee  or  multiple  interview  process, 
and  these  do  not  share  the  advantages  of  the 
Delphi  technique. 


P(X5)  +  P(Xj)  =  0.10  (0.377)  =  0.038 

P(X6)  +  P(Xl)  =  0.0  (0.377)  =  0.000 
RX  i 

P(X7)  +  P(Xj)  =  0.0  (0.377)  =  0.000 
RX  j 


The  Delphi  technique  has  been  extended  in 
recent  years  to  cover  a  wide  variety  of  types  of 
group  interaction.  The  technique  can  be  used 
for  group  estimation,  that  is,  the  use  of  a  group 
of  knowledgeable  individuals  to  arrive  at  an 
estimate  of  an  uncertain  quantity.  The  quantity 
can  be  a  cost,  a  time  period  associated  with  an 
event,  or  a  performance  level. 


The  resulting  probability  density  appears  in 
Table  D-5. 


Component 

Characteristic 

Value 

Probability 

Xi 

0.377 

X2 

0.301 

x3 

0.189 

x4 

0.095 

x5 

0.038 

x6 

0.000 

x7 

0.000 

Total 

1.000 

Table  D-5.  Probability  Density 


5.  Delphi  Approach 

In  many  cases,  expert  judgement  does  not  reside 
solely  with  one  individual,  but  is  spread  among 
multiple  experts.  Committee  approaches  to 
obtaining  a  group  assessment  have  been  found 
to  contain  problems  relating  to  interpersonal 
pressures  to  a  degree  that  caused  researchers  at 
the  RAND  Corporation  to  devise  a  method 


The  Delphi  technique  is  most  appropriate  when: 

•  The  problem  does  not  lend  itself  to  precise 
analytical  techniques  but  can  benefit  from 
subjective  judgements  on  a  collective  basis. 

•  The  individuals  needed  to  contribute  to  the 
examination  of  a  broad  or  complex  problem 
have  no  history  of  adequate  communication 
and  may  represent  diverse  backgrounds  with 
respect  to  experience  or  expertise. 

•  More  individuals  are  needed  than  can  effec¬ 
tively  interact  in  a  face-to-face  exchange. 

•  Time  and  cost  make  frequent  group  meetings 
unfeasible. 

•  The  efficiency  of  face-to-face  meetings  can 
be  increased  by  a  supplemental  group 
communication  process. 

•  Disagreements  among  individuals  are  so 
severe  or  politically  unpalatable  that  the  com¬ 
munication  process  must  be  refereed  and/or 
anonymity  assured. 


D-7 


•  The  heterogeneity  of  the  participants  must  be 
preserved  to  assure  validity  of  the  results,  i.e., 
avoidance  of  domination  by  quantity  or  by 
strength  of  personality  (“bandwagon  effect”). 

The  Delphi  technique  differs  from  other 
methods  of  obtaining  a  group  opinion,  because 
it  physically  separates  the  group’s  members 
from  one  another  in  order  to  reduce  irrelevant 
interpersonal  influences.  Properly  carried  out, 
the  technique  is  facilitated  by  an  analyst 
obtaining  each  panel  member’s  reason  for  the 
opinion.  The  analyst  then  reduces  the  opinions 
and  reasons  to  standard  statements  in  order  to 
preserve  anonymity.  The  analyst  then  shows  the 
panel  member  the  aggregated  opinions  of  the 
other  panel  members  in  statistical  terms.  The 
analyst  provides  each  panel  member  with  the 
reasons  justifying  the  opinions  that  differ  with 
the  member,  and  requests  revaluation  and  fur¬ 
ther  substantiation.  This  iterative  feeding  back 
continues  until  no  further  substantial  change 
results.  At  this  point,  the  moderator  takes  the 
final  individual  opinions  and  computes  a  set  of 
median  values  to  represent  the  group  opinion. 
The  median  value,  rather  than  the  average,  is 
used  as  a  central  estimate  to  prevent  the  esti¬ 
mate  from  being  overly  influenced  by  extreme 
individual  values. 

One  technique  which  holds  much  promise  for 
the  future  as  a  means  of  capturing  expert  judge¬ 
ment  is  “expert  support  systems”.  Ideally,  the 
expert  support  system  would  lead  the  expert(s) 


through  a  series  of  parameter  specific  questions 
(cost  and  schedule,  possible  performance)  and 
generate  PDFs  based  on  the  responses. 

III.  RELIABILITY 

The  reliability  of  the  PDFs  obtained  through 
these  techniques  is  affected  by  a  number  of  fac¬ 
tors.  Foremost  is  the  degree  to  which  the  so 
called  “expert”  is  in  fact  an  expert.  The  better 
understanding  the  expert  has  of  the  parameter 
being  modeled,  the  more  reliable  the  resulting 
PDFs  will  be.  The  burden  also  falls  on  the  ana¬ 
lyst  to  select  the  technique  most  appropriate  for 
obtaining  PDFs.  For  example,  if  expertise 
resides  with  more  than  one  expert,  a  Delphi 
technique  would  result  in  much  more  reliable 
PDFs  than  would  a  direct  method  of  asking  only 
one  expert.  Likewise,  if  the  expert  has  very  little 
understanding  of  probability  concepts,  it  would 
be  inappropriate  to  ask  him  to  select  a  PDF  from 
a  visual  list  of  options.  Under  these  circum¬ 
stances,  the  modified  Churchman- Ackoff 
method  or  a  betting  technique  would  most  likely 
result  in  more  reliable  PDFs. 

In  summary,  much  of  the  reliability  of  the  PDFs 
is  predicated  on  the  techniques  selected  by  the 
analyst  for  constructing  them.  Therefore,  it  is 
important  that  the  analyst  know  when  each 
technique  is  most  appropriate,  given  the  unique 
circumstances  of  that  specific  program  office. 
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